Session:「Collaborative Crowdwork」

CrowdVerge: Predicting If People Will Agree on the Answer to a Visual Question

論文URL: http://dl.acm.org/citation.cfm?doid=3025453.3025781

論文アブストラクト: Visual question answering systems empower users to ask any question about any image and receive a valid answer. However, existing systems do not yet account for the fact that a visual question can lead to a single answer or multiple different answers. While a crowd often agrees, disagreements do arise for many reasons including that visual questions are ambiguous, subjective, or difficult. We propose a model, CrowdVerge, for automatically predicting from a visual question whether a crowd would agree on one answer. We then propose how to exploit these predictions in a novel application to efficiently collect all valid answers to visual questions. Specifically, we solicit fewer human responses when answer agreement is expected and more human responses otherwise. Experiments on 121,811 visual questions asked by sighted and blind people show that, compared to existing crowdsourcing systems, our system captures the same answer diversity with typically 14-23% less crowd involvement.

日本語のまとめ:

ある画像に関する質問への答えが、複数のクラウドワーカー間で一致するかどうかを予測する手法の提案。視覚健常者・障害者により撮影された画像121,811枚について手法を適用し、回答の多様性を保ったまま14-23%のコスト削減に成功した。

Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations

論文URL: http://dl.acm.org/citation.cfm?doid=3025453.3025811

論文アブストラクト: This paper introduces flash organizations: crowds structured like organizations to achieve complex and open-ended goals. Microtask workflows, the dominant crowdsourcing structures today, only enable goals that are so simple and modular that their path can be entirely pre-defined. We present a system that organizes crowd workers into computationally-represented structures inspired by those used in organizations - roles, teams, and hierarchies - which support emergent and adaptive coordination toward open-ended goals. Our system introduces two technical contributions: 1) encoding the crowd's division of labor into de-individualized roles, much as movie crews or disaster response teams use roles to support coordination between on-demand workers who have not worked together before; and 2) reconfiguring these structures through a model inspired by version control, enabling continuous adaptation of the work and the division of labor. We report a deployment in which flash organizations successfully carried out open-ended and complex goals previously out of reach for crowdsourcing, including product design, software development, and game production. This research demonstrates digitally networked organizations that flexibly assemble and reassemble themselves from a globally distributed online workforce to accomplish complex work.

日本語のまとめ:

クラウドワーカーの役割・チーム・階級の概念を持たせた組織を構成し、複雑かつオープンエンドな成果物を生成するプラットフォームの提案。運用実験において、医療用Androidアプリ・カードゲーム・商用ポータルサイトの開発に成功した。

Facilitating Pervasive Community Policing on the Road with Mobile Roadwatch

論文URL: http://dl.acm.org/citation.cfm?doid=3025453.3025867

論文アブストラクト: We consider community policing on the road with pervasive recording technologies such as dashcams and smartphones where citizens are actively volunteering to capture and report various threats to traffic safety to the police via mobile apps. This kind of novel community policing has recently gained significant popularity in Korea and India. In this work, we identify people's general attitude and concerns toward community policing on the road through an online survey. We then address the major concerns by building a mobile app that supports easy event capture/access, context tagging, and privacy preservation. Our two-week user study (n = 23) showed Roadwatch effectively supported community policing activities on the road. Further, we found that the critical factors for reporting are personal involvement and seriousness of risks, and participants were mainly motivated by their contribution to traffic safety. Finally, we discuss several practical design implications to facilitate community policing on the road.

日本語のまとめ:

地域の治安維持を目的とした、容易な映像記録・通報のためのスマートフォンアプリ開発と運用評価を行った。2週間に渡り23人の被験者を募り運用した結果、通報には事案への関与の有無とリスクの深刻性が重要な要素であり、また通報のモチベーションはドライバーとしての交通安全への貢献意識であることが分かった。

ReTool: Interactive Microtask and Workflow Design through Demonstration

論文URL: http://dl.acm.org/citation.cfm?doid=3025453.3025969

論文アブストラクト: In addition to simple form filling, there is an increasing need for crowdsourcing workers to perform freeform interactions directly on content in microtask crowdsourcing (e.g. proofreading articles or specifying object boundary in an image). Such microtasks are often organized within well-designed workflows to optimize task quality and workload distribution. However, designing and implementing the interface and workflow for such microtasks is challenging because it typically requires programming knowledge and tedious manual effort. We present ReTool, a web-based tool for requesters to design and publish interactive microtasks and workflows by demonstrating the microtasks for text and image content. We evaluated ReTool against a task-design tool from a popular crowdsourcing platform and showed the advantages of ReTool over the existing approach.

日本語のまとめ:

クラウドソーシングのリクエスタ向けの、タスクでの作業内容をウェブ上で再現(実演)することでワークフロー生成を容易に行えるツールの開発。Mechanical Turk上で生成する従来の方法と比較し、実用性を示した。