Job Description
SR ANALYTICAL LINGUIST, HUMAN COMPUTATION
WE’RE LOOKING FOR TWITTER’S VERY FIRST ANALYTICAL LINGUIST TO JOIN OUR HUMAN COMPUTATION TEAM.
TWITTER SERVICE IS ONE OF THE PREMIER SCALED SERVICE ORGS WITHIN TWITTER. THE SERVICES WE PROVIDE RANGE FROM ENFORCEMENT OPERATIONS, TO HUMAN COMPUTATION & LOCALIZATION. OUR MISSION IS TO PROVIDE MEANINGFUL AND EFFECTIVE SERVICES TO OUR CUSTOMERS, BE IT INTERNAL OR EXTERNAL.
THE HUMAN COMPUTATION TEAM’S MISSION IS TO STREAMLINE PRODUCTION OF HIGH-QUALITY LABELED DATA AND CONTEXT-BASED INSIGHTS & MEASUREMENT, IN SUPPORT OF MACHINE LEARNING, OFFLINE METRICS, AND ALL OTHER HUMAN-IN-THE-LOOP INITIATIVES ACROSS TWITTER. OUR WORK DIRECTLY IMPACTS THE QUALITY AND CONSUMPTION OF OUR MOST POPULAR PRODUCTS, HELPING TO PREVENT THE ABUSE AND MISUSE OF TWITTER, AND DELIGHTING OUR EVER EXPANDING GLOBAL USER BASE.
THE IDEAL CANDIDATE IS SOMEONE WHO HAS A PROVEN TRACK RECORD OF DESIGNING AND BUILDING COMPLEX HUMAN EVALUATIONS AND EXPERIMENTS, IN SUPPORT OF DATA ACQUISITION/ANNOTATION AND ERROR ANALYSIS PROJECTS FOR ML & MORE. THEY WILL HAVE EXTENSIVE EXPERIENCE IN 1) WRITING CLEAR AND CONCISE HUMAN EVAL GUIDELINES (PREFERABLY WITH PREVIOUS EXPERIENCE DRIVING COMPLEX AND SUBJECTIVE AREAS LIKE ONLINE MISINFORMATION, TOXICITY, AND/OR OTHER NLP AREAS LIKE CONTENT UNDERSTANDING/TOPIC DETECTION ETC), 2) CONDUCTING IN-DEPTH DATA ANALYSIS TO IDENTIFY ERRORS AND PATTERNS IN DATA, AND USING THOSE TO ITERATE AND IMPROVE QUALITY OF THE LABELED DATA AND INSIGHTS. 3) COACHING AND MENTORING JUNIOR ANALYTICAL LINGUISTS TO BE SUCCESSFUL. THE RIGHT PERSON WOULD ALSO HAVE EXPERIENCE WORKING WITH MULTIPLE GLOBAL STAFFING VENDORS AND EFFECTIVELY NAVIGATING A CHALLENGING, FAST-PACED, AND DYNAMIC DAILY WORKING ENVIRONMENT.
RESPONSIBILITIES
DRIVE DATA ANNOTATION PROJECTS TO BUILD AND TEST THE QUALITY OF MACHINE LEARNING MODELS, TO GENERATE VARIOUS OFFLINE MEASUREMENT INITIATIVES, AND TO CONDUCT CONTEXT-BASED ANALYSIS IN SUPPORT OF FEATURE LAUNCHES AND MORE.
IDENTIFY ERRORS AND REGRESSIONS BY LEVERAGING HUMAN-IN-THE-LOOP EXPERIMENTS AND DATA ANALYSIS, AND PROPOSE SOLUTIONS TO IMPROVE MODEL ACCURACY AND PERFORMANCE.
WORK CLOSELY WITH THE MACHINE LEARNING ETHICS AND ACCOUNTABILITY TEAM TO IDENTIFY, MEASURE, AND MITIGATE BIAS & OTHER ETHICAL ISSUES THAT MIGHT AFFECT ML MODELS USING HUMAN LABELED DATA.
DRAFT AND MAINTAIN CLEAR AND CONCISE HUMAN EVALUATION GUIDELINES, BY COLLABORATING WITH ENGINEERING, PRODUCT, RESEARCH, AND OTHER PARTNERS.
BUILD EVALUATION USER INTERFACES USING JAVASCRIPT, HTML, AND CSS. BE PROFICIENT IN BASIC DATA ANALYSIS USING PYTHON, R, OR OTHER DATA ANALYSIS TOOLS.
PROPOSE QUALITY MEASURES AND DESIGN QUALITY CONTROL METHODOLOGY TO OPTIMIZE QUALITY AND THROUGHPUT OF DATA ANNOTATION PROJECTS.
RUN EXPERIMENTS TO TEST VARIOUS EVALUATION ATTRIBUTES (SIZE, FORMAT, GUIDELINES, QUESTION FRAMING+ORDER, STAFFING MODELS ETC) TO OPTIMIZE FOR QUALITY AND THROUGHPUT OF LABELED DATA.
IDENTIFY AND PROPOSE SIMPLE TECHNICAL SOLUTIONS AND DATA ANNOTATION SYSTEMS, TO ENABLE AUTOMATED AND BATCH LABELING MECHANISMS.
DELEGATE LABELING TASKS ACROSS INTERNATIONAL AND MULTI-VENDOR WORKFORCE SOLUTIONS, IN ORDER TO CONSISTENTLY EXCEED THE EXPECTATIONS OF OUR INTERNAL BUSINESS PARTNERS.
REVIEW AND PROVIDE FEEDBACK ON TECHNICAL DESIGN DOCUMENTS FROM ENGINEERING AND PRODUCT TEAMS, WITH RESPECT TO HUMAN COMPUTATION’S SCOPE.
Qualifications
EDUCATION
LINGUISTICS AND HUMAN COMPUTATION EXPERIENCE
MINIMUM 5+ YEARS OF EXPERIENCE WITH HUMAN COMPUTATION/HUMAN-IN-THE-LOOPS SYSTEMS, LARGE SCALE DATA EXTRACTION AND ANALYSIS, ML/AI DATA STRATEGY & OPERATIONS, AND CROWDSOURCING.
MINIMUM 5+ YEARS OF EXPERIENCE DRAFTING HUMAN EVALUATION GUIDELINES AND REFINING THOSE BASED ON EXPERIMENTS & RESEARCH.
MINIMUM 4+ YEARS EXPERIENCE AS AN ANALYTICAL/COMPUTATIONAL LINGUIST DRIVING DATA ANNOTATION/CURATION EFFORTS AND/OR ERROR ANALYSIS PROJECTS, IN SUPPORT OF MACHINE LEARNING.
A GOOD UNDERSTANDING OF DIFFERENT TECHNIQUES USED TO MITIGATE AND REDUCE THE EFFECTS OF CONSCIOUS AND UNCONSCIOUS BIAS IN LABELED DATA.
FAMILIARITY AND EXPOSURE TO MACHINE LEARNING CONCEPTS, AND A DEEP UNDERSTANDING OF HOW LABELED DATA AND HUMANS-IN-THE-LOOP CAN BE LEVERAGED TO IMPROVE ML SYSTEMS.
DATA & OTHER TECHNICAL EXPERIENCE
OVER 4 YEARS EXPERIENCE UTILIZING SQL, SCALDING, R AND PYTHON AND OTHER PROPRIETARY DATA EXTRACTION AND ANALYSIS TOOLS.
OVER 4 YEARS EXPERIENCE IN DESIGNING LABELING USER INTERFACES USING HTML, XML, CSS, JAVASCRIPT, AND EXPOSURE TO LEADING LABELING PLATFORMS LIKE FIGURE-EIGHT/APPEN ETC.
OVER 4 YEARS OF EXPERIENCE IN ANALYZING LARGE DATASETS TO DERIVE INSIGHTS, IDENTIFY PATTERNS & ERRORS IN LABELED DATA, AND IN TRANSLATING THOSE INTO TANGIBLE GUIDELINES AND USER INTERFACE DESIGN IMPROVEMENTS.
OTHER REQUIREMENTS
MINIMUM 3 YEARS EXPERIENCE COACHING, MENTORING, AND HIRING JUNIOR MEMBERS OF THE TEAM. PRIOR PEOPLE MANAGEMENT EXPERIENCE IS A PLUS, BUT NOT A REQUIREMENT.
ABILITY TO INFLUENCE STAKEHOLDERS TO ACHIEVE COMMON GOALS AND TO PARTNER WITH VARIED XFN TEAMS TO FURTHER COMPANY/TEAM GOALS.
ABILITY TO WORK INDEPENDENTLY, WITH MINIMUM GUIDANCE, AND HELP/COACH OTHERS ON THE TEAM TO GET BETTER RESULTS.
DEMONSTRATED ANALYTICAL SKILLS, EXCELLENT WRITTEN AND VERBAL COMMUNICATION SKILLS IN A TECHNICAL CONTEXT, STRONG ORGANIZATIONAL SKILLS, AND VERY DETAILED ORIENTED.
ACTIVE, PASSIONATE TWITTER USER WITH A STRONG INTEREST IN SOCIAL MEDIA. GOOD GRASP OF POP CULTURE, SPORTS, AND OTHER TOPICS THAT ARE CONVERSATION DRIVERS ON TWITTER.
PREFERED QUALIFICATIONS
MA, MS, OR PHD DEGREE IN COGNITIVE SCIENCE, PSYCHOLOGY, SOCIAL SCIENCE, COMPUTER SCIENCE, OR EXPERIENCE IN A RELATED FIELD.
OVER 6+ YEARS OF EXPERIENCE WITH HUMAN COMPUTATION/HUMAN-IN-THE-LOOPS SYSTEMS, LARGE SCALE DATA EXTRACTION AND ANALYSIS, AND CROWDSOURCING.
5+ YEARS EXPERIENCE AS AN ANALYTICAL/COMPUTATIONAL LINGUIST DRIVING DATA ANNOTATION AND ERROR ANALYSIS PROJECTS IN SUPPORT OF MACHINE LEARNING.
PREVIOUS EXPERIENCE IN QUALITATIVE AND QUANTITATIVE ANALYSIS, RESEARCH AND SURVEY METHODOLOGIES, AND OTHER TECHNIQUES TO LEVERAGE HUMAN COMPUTATION IN SUPPORT OF VARIED USE CASES.
ACTIVE, PASSIONATE TWITTER USER WITH A STRONG INTEREST IN SOCIAL MEDIA. INTEREST IN TWITTER’S PRODUCTS AND MISSION, AND ABOUT THE ROLE MACHINE LEARNING COULD PLAY TOWARDS ADVANCING THEM.
Additional Information
We are committed to an inclusive and diverse Twitter. Twitter is an equal opportunity employer. We do not discriminate based on race, ethnicity, color, ancestry, national origin, religion, sex, sexual orientation, gender identity, age, disability, veteran, genetic information, marital status or any other legally protected status.
San Francisco applicants: Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records