## 1. Project Overview & Quickstart (overview/compatibility) # compatibility Open-source repository overview/compatibility ### Repository Details - **Repository:** [overview/compatibility](https://github.com/overview/compatibility) - **Primary Language:** Code *Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.* ## 2. Official Technical Reference & Guides (overview/overview.github.io) ## File: docs/case-studies/analyze-social-media-posts.md --- title: How to use Overview to analyze social media posts parent: Case Studies grand_parent: Overview Help nav_order: 1 --- # How to use Overview to analyze social media posts Even when 10,000 people post about the same topic, they’re not saying 10,000 different things. People talking about an event will focus on different aspects of it, or have different reactions, but many people will be saying pretty much the same thing. People posting about a product might be talking about one or another of its features, the price, their experience using it, or how it compares to competitors. Citizens of a city might be concerned about many different things, but which things are the most important? Overview’s document sorting system groups similar posts so you can find these conversations quickly, and figure out how many people are saying what. This post explains how to use Overview to quickly figure out what the different “conversations” are, how many people are involved in each, and how they overlap. __Step 1: Get the data into Overview__ Overview does not capture or scrape social media. Fortunately, there are lots of other monitoring tools to do that, such as Radian 6, Sysomos, and Datasift. Overview can read data exported from each of these tools as CSV files. This is a simple file format used by spreadsheet programs, so if you can open it in Excel, you should be able to import it to Overview. Or, you can create your own files in this format. Then upload the file into overview. Select “Import a New Document Set” as usual, then choose “CSV upload” and select your file. Overview will check the file for errors and show a preview of the file, like this: If it all looks good, click Upload. __Step 2: Explore your documents__ Overview reads the full text of each post — not just names and keywords — and groups similar documents together into different folders. Then it takes each folder, and groups similar documents into sub-folders, and so on. The folder tree shows how Overview has organized your posts. There won’t always be strictly one type of “conversation” per folder, because conversation is a fuzzy concept, and Overview doesn’t know how you want to categorize things. But all the documents in the folder will be the same in some way — or if the folder seems to include too many different types of posts, try exploring the sub-folders, which have narrower topics. As you select each folder, Overview shows a list of the posts in that folder in the bottom left. Each post is summarized by the most “characteristic” words in that post — not the words that are most common, but the words that make the documents in that folder different from the documents in other folders. It’s the tags you create that define which conversations you’re interested in. These might be very broad categories such as “positive” and “negative,” or much more specific tags such as “didn’t like the color.” Generally, you’ll explore the documents top-to-bottom and left-to-right, inventing and assigning tags as you go. You can assign tags to one document at a time if you like, but usually you’ll want to assign a tag to all documents in the folder at the same time. To do this, select the folder, then move the mouse over the tag and press the + button. Tags are non-exclusive, meaning that each document can have as many tags as you like. __Step 3: Export your data__ Now that you’ve seen what’s in your data, and tagged it for future reference, you probably want to use these results in some way. You can see how many documents have each tag just by selecting that tag. Or, you can export your entire document set, including the tags you just added. The export button is on the document set list page. It creates a spreadsheet (CSV file) and gives you a choice how you want your tags. Each document will be one row in the speradsheet. If you put all tags in one column they will be separated by commas, like “Sunshine, Positive.” But putting each tag in its own column makes it easy to do things like find all the documents that have a specific tag, or take column totals to get the number of documents with each tag — useful for making a visualization of your results, like this visualization of 16,000 tweets about drones (created with Overview by Tempero UK): --- ## File: docs/case-studies/topic-tree.md --- title: Topic tree - Organize documents into folders parent: Case Studies grand_parent: Overview Help nav_order: 2 --- # Topic tree - Organize documents into folders Before computers, all document-driven stories started with a big stack of paper. Often, the first task was to organize all that paper, by sorting individual documents into piles by type. This gives journalists a high-level idea of “what’s in there” and helps them decide what to read more closely — and just as importantly, what isn’t worth reading. Today a computer can organize your documents for you. That stack of paper may now be a folder full of PDF files, but it still doesn’t come with any sort of built-in index or obvious categorization system. This is exactly the problem that Overview solves: It splits documents into piles based on their subjects, and then splits each pile into even more specific sub-piles, and so on. The result is a tree of folders. Above is part of the folder tree that Overview automatically built for the 6,849 documents containing every mention of the city of “Caracas” within the diplomatic cables released by Wikileaks. (Click on the image for a larger version.) Overview labels each folder by the key words in the documents inside. The top folder here has words like PDVSA (the Venezuelan state oil company), “oil”, “billion,” “company” and “production,” so it’s mainly documents concerning the oil industry and other big business. Other top-level folders in this document set (not shown) concern embassy politics, elections, and military operations. The top-level folder about oil splits into two sub-folders. The one on the left concerns the oil industry specifically, while the one on the right is more about banks and finance. The oil industry folder splits further into regional issues (the Petrocaribe consortium) and documents about PDVSA specifically. Each folder splits into smaller and smaller sub-folders, each of which contains a smaller number of documents on a more specific topic. To let you know when the documents in a folder are getting very specific, Overview tells you when “MOST” or “ALL” of the documents in that folder contain a particular word. ## How a computer understands topics When I show this to reporters, their first question is always, how does the computer do that? It’s more than curiosity: If you’re going to rely on a computer to organize your documents, you’re asking a machine to help you decide what you should and shouldn’t read. The integrity of the reporting process demands that we understand what our algorithms are doing. All document categorization algorithms are based on the ability to compare two documents to tell how similar they are. A group of documents which are all very similar to one another belong in the same folder. Computers don’t understand human language, so they need a simple mechanical process which takes two documents as input — literally just the sequence of words that make up the text of each document — and generates a number which is small if the documents are very different, and large if the documents concern the same topic. Some text analysis systems, such as Open Calais, are based on “named entity recognition,” which extracts people, places, organizations, dates, etc. from the documents. Then, we can say that two documents are similar if they talk about the same entities. This is useful, but such systems will miss important generic words like “oil” and “production.” Instead, Overview examines every word of every document. In a sense, it reads the full text, so you don’t have to. ## Comparing two documents based on their full text Suppose we have filed an FOIA request for a classified storybook for the children of CIA operatives, and after a long legal battle. The government has given us copies of these three secret documents: * “The cat sat on the mat. Then the cat chased the rat.” * “The cat slept all day on the mat.” * “The rat ran across the floor.” First, Overview strips capitalization, punctuation, and the grammar words such as “the,” “a,” “on,” etc. These words, also called stop words in natural language processing, aren’t useful for determining the topic of the text, because they appear in almost every document. This leaves us with: * “cat sat mat cat chased rat” * “cat slept all day mat” * “rat ran across floor” You can see that most of the sense of the document is still there, despite removing lots of words. Then, Overview counts how many times each word appears in each document, producing a word frequency table, like this: This throws out the order of the words, which means the computer can’t understand the difference between “soldiers shot civilians” and “civilians shot soldiers.” This may seem very simplistic, but surprisingly, decades of information retrieval research show that word order usually doesn’t matter when all you want to know is the topic of a document. Then Overview compares every pair of documents to check how similar they are. It does this by counting the number of words which appear in both documents, but with a twist: If a word appears twice in one document, it’s counted twice. In other words, we multiply the frequencies of corresponding words, then add up the results. This is the final similarity score. In this case, the two documents about the cat have a similarity of 3: Cat appears twice in the first document and once in the second, plus rat appears once in each document. The document about the rat has no words in common with the document about the cat sleeping on the mat, so the similarity score is zero. Documents which are similar enough end up in the same folder, and the folder is labelled by the words which make those documents different from all the others. In this case, the folder is labeled by “cat” and “mat” because those words don’t appear in the remaining document about the rat. And that’s the heart of it. This description omits a number of details for simplicity, but includes all of the things a reporter needs to know: * Overview uses the full text of each document. * It is not sensitive to word order. * Documents with overlapping words are placed in the same folder. If you’d like to understand the process more deeply, here are a few more details: Overview actually processes text in two word bigrams, not just single words, so it can detect people’s names and other short phrases. Rather than just simple term counts, it weights each word by how rare it is in the document set overall, using a classic formula called TF-IDF. And to generate the folders, given the similarity between every pair of documents, Overview uses k-means clustering, splitting folders recursively at each level of the tree. ## Try it on your own documents Overview can automatically import your projects from the popular DocumentCloud repository, which also handles document upload, OCR, and other tasks. Or, you can upload a CSV file if your text is already in spreadsheet or database format. It also works great on social media data, such as a collection of tweets or blog posts. --- ## File: docs/custom-development/api-and-plugins.md --- title: Overview APIs and plugins parent: Custom Development grand_parent: Overview Help nav_order: 7 --- # Overview APIs and plugins For information about Overview APIs and plugins, please consult the [API Overview](https://github.com/overview/overview-server/wiki/Overview-API-Overview) --- ## File: docs/custom-development/contact.md --- title: Contact parent: Custom Development grand_parent: Overview Help nav_order: 3 --- # Contact Need help? Questions? Bugs? Interested in support or custom features for your business? We really want to hear from you. Email us at __info@overviewdocs.com__ --- ## File: docs/custom-development/importing-from-command-line.md --- title: Importing from the command line parent: Custom Development grand_parent: Overview Help nav_order: 2 --- # Importing from the command line For information about importing from the command line, please consult the [Overview Upload Directory](https://github.com/overview/overview-upload-directory) --- ## File: docs/custom-development/index.md --- title: Custom Development has_children: true nav_order: 7 --- # Custom development If you have a specific need that you do not see addressed here, please don't hesitate to contact us. We can steer you in the right direction, and even develop custom solutions to fit your exact needs. Email us at __info@overviewdocs.com__ --- ## File: docs/export/export-to-excel.md --- title: Exporting to Microsoft Excel parent: Exporting grand_parent: Overview Help nav_order: 1 --- # Exporting to Microsoft Excel Overview has long had a fairly important (if little-used) feature: it can export all documents into a spreadsheet. Day one, we wrote the spreadsheet in “comma-separated values” (“CSV”) format. Then we realized that Microsoft Excel couldn’t open all CSV files. Day two, we implemented a separate export file type, just for Microsoft Excel. Here are the differences. ## We truncate document text One of the columns in our exported spreadsheet contains the entire text of each document. Microsoft Excel limits cell size to 32,767 characters. If Excel encounters a document with more than 32,767 characters of text, it stops importing. (We’ve found that most documents in Overview are far shorter than 32,767 characters. For a sense of scale: The Adventures of Huckleberry Finn is 610,000 characters; its average chapter is about 14,000 characters.) When you choose Excel format, Overview truncates the text of very-long documents. ## We avoid Excel for Mac file-encoding issues We spent ages debugging this issue, because we had to quell our disbelief. It turns out the “Encoding” option in Excel for Mac 2011’s CSV-import dialog doesn’t do anything. If you import an Overview CSV into Microsoft Excel for Mac 2011, Excel will garble those special characters. The same holds for virtually any CSV on the Internet that has special characters such as “smart quotes”. (Techno-lingo: Excel for Mac does not decode UTF-8, even though it presents the choice to decode UTF-8.) We don’t know whether Microsoft solved this issue in Excel for Mac 2016. Overview doesn’t do anything special to overcome this problem: it’s only a bug with the CSV-import feature. We recommend you avoid importing any CSVs into Microsoft Excel for Mac 2011 unless you’re certain they aren’t encoded in UTF-8. (UTF-8 is the most common encoding on the Internet.) ## The take-away If you use Microsoft Excel, pick Excel format when you export from Overview. Overview will give you a file that’s as close to the original data as possible, and Excel will be able to open that file. If you use a different spreadsheet program, pick CSV format for an _exact_ copy of the data within Overview.