One of the main goals of analyzing clinical data is to produce a report. (What, you thought it was to make the world a better place?) The R Project has, of course, all the tools you need to perform the statistical analysis, calculate the tables of results, and present conclusions graphically. But how can you assemble all of that into a report that someone can, you know, read? You could go the cut-and-paste route: write the text in Word, export the data from R to format the tables in Excel, dress up the saved charts in Photoshop. But that's a complex, manual process, and manual processes can introduce errors. Worse yet, if the data ever changes, you've got to go through the whole process again to update the report. That means no interim reports, and conversely, a big barrier to correcting the data and the report after it's published. Vanderbilt Biostatistics professor Frank Harrell has a different solution: the rreport package for R. (See an overview slide deck here.) It's designed to produce statistical reports for clinical trials... read more >>
|
In a previous post last year I mentioned a particularly nasty and blatant breach of confidentiality regarding fixed line telephony data. The update is that Optus recently won a court case in Federal Court to seek damages against Telstra; http://www.itnews.com.au/News/168876,optus-wins-telstra-confidentiality-breach-ruling.aspx This news seemed to slip the major national newspapers, which is quite surprising as it is likely to involve significant amounts of money. To be honest I’m not concerned with the consequences, but as a data miner it does interest me how data *is* used, and how it *could* be used. As technology advances I’m certain the general public will see more examples of invasions of personal privacy and breaches of data confidentiality... read more >>
|
Stealth buzz: how Lyza stole the show Lyzasoft wasn’t among the 38 exhibitors in TDWI’s Las Vegas exhibit hall. They sponsored no part of the lunch, and they hired no stage magician. But their buzz was the loudest I heard over the event’s five days. Others may have heard different buzz because buzz varies. Business intelligence elites gather every year at TDWI’s big Las Vegas event to teach, and they end up schmoozing, too. Over beer, food, and sometimes playing cards, they compare notes. Is anyone seeking a consensus? I suppose someone might, but the interesting ones just play with ideas, reflect on what others say, make a joke, and think about it. If there’s any “truth,” it develops during a lot of talk and thought, whether it’s about politics, tofu, the future of passenger rail in America, or business. That goes for any kind of conversation, whether the medium is words or data. In business, the conversation is somehow forgotten in favor of the data. But to Scott Davis, CEO of Lyzasoft ... read more >>
|
John Reynolds had an interesting post a little while back where he shared some thoughts on Perfect development tools. His emphasis was on support for things like iterative and test-driven development but it seems to me that there is also a need to move application development beyond code. While developers do need development environments that support new approaches to developing code that works and that help speed and improve the application development process, they also need tools that help them move beyond code. In particular they need a development environment that: - Ensures that decision-making logic is managed declaratively as a set of business rules
- Integrates analytics into this decision-making logic in a useful way
- Helps them put process or workflow into a process management tool
- Helps them define events and how events will be correlated and processed
As long as development environments assume that everything can and should be written as code, I do not believe they will be “perfect”. Code is the right way to do a whole bunch of things in application development... read more >>
|
Recently, The Economist had a special report titled “Data, data everywhere“. The report examines the rapid increase in data volumes and what the implications are. The report got the attention of the blogosphere (example) and I recommend taking a look if you haven’t already. When I read articles like these, I try to extract three categories of “knowledge” for future use: factoids, stories, and insights. - Factoids are simply data points that I feel might come in handy someday
- Stories are real-world anecdotes. The most memorable ones have an “aha!” element to them.
- Insights are observations (usually at a higher level of abstraction than stories) that make me go “I never thought of that before. But it makes total sense.”
Think of this crude categorization as my personal approach to dealing with information overload. Of course, there’s a fair amount of subjectivity here: what I think of as an insight may be obvious to you and vice-versa. So what did I make of The Economist article? There were numerous factoids that I cut-and-stored away (too many to list here but email me if you want the list)... read more >>
|
Well, James Cameron didn’t exactly walk away empty-handed from Sunday’s 2010 Academy Awards, winning three Oscars in predominantly technical categories out of nine nominations. But, I’d be willing to bet he expected an even-bigger armful of the trophies, especially considering that Avatar is THE biggest box-office success in the world, ever. While the awards weren’t necessarily forthcoming, I suspect the Academy still recognizes a revolution when it sees one. The true advent 3-D content has arrived—and Avatar has made that very clear in a very big way.
I remember a time, something like three decades ago, when a 3-D experience meant a trip to the local 7-Eleven to buy green and red lenses to watch an Elvira “Movie Macabre” horror special on our home television. The highlight came as the Mistress of the Dark tossed a few handfuls of popcorn at the screen, causing me to squeal in delight. The fact that my parents allowed their young child to watch such garbage is the subject of another discussion.
My how things have changed. It’s been more than three-decades since 3-D made its first forays into our living rooms... read more >>
|
In “Chapters from My Autobiography”, Mark Twain wrote, “There are lies, damned lies, and statistics.”[1] While over one hundred years old, the maxim still applies today. I have been thinking quite a bit about that quote. You see, I’ve been searching for a sponsor for my podcast and one of the first questions that potentially interested companies have is, “How many people listen to each episode?” This is a very understandable request but one not easily answered. Why is such a simple question so vexing? Now, I know from my Wordpress statistics that a certain number of unique visitors stop by blog every day. That number tends to be a little north of 100 on weekdays and, for some reason, wildly higher and lower on some days. Who knows why? But what about total hits? Well…here’s where it gets more complicated. In this post, I look at some of the reasons that your site stats may not reflect your total readership and, ultimately, influence... read more >>
|
In the world of broadband, South Korea is a titan, with top data speeds and market penetration, both for fixed line and wireless. Yet in a panel discussion at the Abu Dhabi Media Forum, Suk-Chae Lee, chairman of Korea's KT Corporation, warned of a data logjam. 'There are limits to the mobile network' he said. Traffic surrounding big news events, such as the suicide of former president Roh Moo-hyun, nearly paralyzed KT's network. He predicted that the limits of wireless networks could strengthen the hand of the same carriers that are failing to build robust networks. With shortages, they might be able to start billing for megabytes, instead of the more common all-you-can-eat subscriptions. 'The dreams of the wireless phone companies will be realized' he said. The trouble is that handsets, like Apple's iPhone, are picking up more of the data work from laptops. Lee mentioned on of KT's customers, a gaming company, that gave iPhones to employees. In the first month, 60% of them moved all computing to their handsets and never even booted up their PCs. 'Will the network sustain that heavy traffic?' he asked ... read more >>
|
White papers are still one of the most popular marketing tools for BI and high tech companies, among others. Proof positive is that I just finished my fifth white paper in almost as many weeks. I find some long held beliefs about white papers very interesting. Like many opinions, some are held long past their expiration dates. For some, they want things a certain way because “that’s the way we’ve always done it.” I particularly find strong opinions around whether or not to include a table of contents and executive summary, and how long the white paper should be. Here are some considerations I think should help make an objective decision: Table of contents – Personally, I’m not a big fan of using a ToC in white papers, unless they are very long (12 or more pages). Your goal is to get your reader reading right away and keep them reading. The ToC is yet one more distraction. Having said that, I think they are very helpful in longer papers. Readers can use them to quickly find sections they want to read again. Skimmers will use them to review the flow to get a quick understanding of where the topic is going... read more >>
|
The R Project: despite there being over 2 million users of this open-source language for statistical data analysis, you might not have heard of it ... yet. You might have seen this feature in the New York Times last year, and you might have heard how REvolution Computing is enhancing and supporting R for commercial use. Because what was once a secret of drug-development statisticians at pharmaceutical companies, quants on Wall Street, and PhD-level statistical researchers around the globe (not to mention pioneers at Web 2.0 companies like Google and Facebook) is suddenly becoming mainstream. The reason? The perfect storm of a deluge of data, open-source technology, and the rise of predictive analytics. Predictive analytics -- the process of being able to infer meaningful relationships and predictions from vast quantities of data -- is disrupting industries in every sector. You've probably seen the impact of predictive analytics yourself: ever been surprised by Amazon apparently "reading your mind" on a suggested purchase, or by LinkedIn being able to figure out who you know, but aren't yet connected with? That's predictive analytics in action... read more >>
|