Archive for the ‘Geeky Shtuff’ Category
After deciding some months ago that none of the currently available e-readers could meet all my needs, I didn’t think about them for a while. The recent “price war” between Amazon and Barnes and Noble has me reconsidering: at less than $200 for a Nook or a Kindle, I could probably live without some of the hardware features I want, especially if there is software that can help bridge the gap.
I’m still undecided about buying a device, but I wanted to catalog some of the programs and hacks that I’ve (re-)discovered as I looked into the issue a second time.
- Calibre looks like a great piece of desktop software for managing e-books. It knows how to talk to multiple e-readers, and can inter-convert between popular formats, allowing (e.g.) Kindle readers to read ePub books by first converting them to Mobipocket format.
- Savory extracts some of the code used by Calibre to allow Kindle users to download and convert ePub books directly on their reader, without having to go through desktop software.
The main issue I have, though, is in dealing with PDFs. It doesn’t look like I’m going to be able to justify the expense of a large-format e-ink reader in the near future (the Kindle DX is the cheapest, I think, at $489!). I’m still looking for a comprehensive set of tools for manipulating PDFs so that I could read them easily on a smaller screen. Specifically, I need tools for:
- extracting text from text-based PDFs, or at least being able to reflow them and trim their margins
- converting scanned images of book pages in PDFs into text via OCR software
I haven’t found a complete solution for either task, but I have come across various programs that do some of these things:
- PDFMunge is a Python program that can help with the task of trimming margins and reflowing text in text-based PDFs.
- pdftk is a comprehensive Java library and command line tool for manipulating PDF files.
- Google Docs now has the option to use OCR to convert PDFs to text. It doesn’t work perfectly, especially for more technical material, but it’s easy to use. I believe it is based on Ocropus and/or tesseract-ocr, both of which are Free software and can be built and run locally (if you can figure out how to do so…the dependencies are pretty significant).
- Briss looks like a nice way to crop scanned PDFs using a GUI interface.
Lately, I have been intrigued by the prospect of buying an e-reader, mostly because I find myself printing and carrying around an enormous amount of PDFs. As a graduate student, I have to read quite a lot, and it would be great if I could keep all my readings in one place, with notes, in a searchable format. I don’t much like reading on a computer screen, so a reader with an e-ink display seems like it would be a great solution for me.
Sadly, none of the e-readers available today seem to have the full set of features I would want:
- e-Ink Display: I can’t read for long periods on an LCD, so that rules out something like a smartphone, netbook, or tablet PC.
- Expandable storage: one of the big downsides of the current Kindle is that its storage is limited to the 1.4GB available to you when the device ships. I especially can’t understand why Amazon removed the SD card slot that the Kindle 1 had.
- Physical keyboard and note-taking abilities: this tells in favor of the Kindle, but against the Barnes and Noble Nook, as well as against a lot of the other e-readers I have seen. I want tactile feedback when I’m typing; I can’t stand typing on touchscreens. Some readers appear to offer no textual input at all, which isn’t much use to me; I need to mark text as I read.
Amazon reserves the right to discontinue wireless connectivity at any time or to otherwise change the terms for wireless connectivity at any time, including, but not limited to (a) limiting the number and size of data files that may be transferred using wireless connectivity and (b) changing the amount and terms applicable for wireless connectivity charges.
Having a fully Internet-ready e-reader would make it much more useful to me, but it’s clear that Amazon wants nothing of the kind. They want to control the kind of information I can get, which they couldn’t do if they included a network interface that — horrors! — didn’t route all traffic through their servers. I don’t really want Amazon, or any other company, knowing everything I read online. And I don’t trust them not to “discontinue…or otherwise change the terms” of my Internet access through their blessed portal.
- Support for open formats, including PDF and ePub: most of the non-Kindle readers win here again, though it’s not clear how much of the PDF standard any of them supports. A lot of the PDFs I read are scanned images from actual books, and I would like to have simple tools for cropping pages, or splitting one page into two, to better fit a reader’s screen.
- Extensible platform: I’d like to be able to write my own programs, or download others’ from the Internet, if the built-in software doesn’t cut it — preferably without having to root the device. For doing academic reading, programs like a multi-lingual dictionary or a citation database would be helpful. Amazon has a Kindle Development Kit in the works, which is nice, except that it’s Java-based; I would much prefer, say, a combination of Python and C. (I’m not sure yet if JVM implementations of Python, Ruby, Scheme, etc. will work on the Kindle…but that would be great!) The Nook has nothing so far, but the fact that it’s running Android points to hackability in the future, with or without Barnes and Noble’s support. Other readers have more explicitly open software platforms, but without a large number of users, they probably won’t see much development.
- Low price: the $259 that Amazon and Barnes and Noble are currently both charging is about as high as I would be willing to go. I simply can’t afford to sink $300 or more into a highly specialized device. This unfortunately rules out a lot of the lesser-known readers for me, because they don’t have the agreements with publishers that would allow them to subsidize their hardware with e-book sales.
So what am I to do? I’d love to be proven wrong, and find a reader out there that has all these features and more. But until then, I think I’m stuck with paper.
One of the frequently-touted benefits of using Free software is that users are not helpless: if a program doesn’t work, or doesn’t do quite what they need it to do, they can either fix the problem themselves (because they have the source code) or they can ask someone else to fix it, possibly for a fee. Users with the same needs can pool their resources to ensure that those needs are met. No user is dependent on a specific developer or company to make the changes they need.
In practice, it is difficult for individual users to exercise this freedom. This is because:
- users may have no idea who to ask to fix a problem or add a new feature; given the nature of most Free software projects, finding the right programmer might be difficult
- even if a user knows who to ask, the programmer or development community might be unwilling to fix the problem because they don’t see it as worth their time
- if the programmer or development community offers to fix the problem or add the feature for a fee, the user may not be able to afford it herself, and may have no idea how to find other people who are also willing to pay for the feature or bugfix
So, although users of Free software have an important freedom in theory, they are still often unable to reap some of the benefits that Free software promises, such as avoiding dependence on a particular developer, or pooling their resources to support development.
This may be one reason why users of popular desktop systems like Ubuntu are often perfectly happy to install proprietary software: even if they know the advantages of using a Free program, without a way to exercise their freedoms, they might as well use a proprietary program that works better or has commercial support.
Meanwhile, because programmers don’t often have a way of getting paid by users to write or adapt Free code, there is a common perception that the only way to earn a living writing software for the masses is to make it proprietary, or to be hired by a large company that will pay you to write software they need. It is difficult for programmers to earn money just by contributing to the Free software projects they are most interested in; sometimes, they ask for donations, but I doubt that many of them expect to earn much from them. We continue to see proprietary programs offered for Free systems as a result. Ubuntu’s Software Center is the latest incarnation of a solution to this perceived problem: Canonical plans, eventually, to sell proprietary applications alongside Free programs as a way of earning money to support their organization. They also think it will attract more programmers to GNU/Linux generally. I hope to be corrected, but I don’t see any part of their proposal that will alleviate the bad choice developers currently face between writing Free code and earning money. They aren’t proposing a payment system for Free software development.
This problem seems eminently solvable. I think we can and should fix the problem of allowing users to exercise their freedoms, and helping developers get paid to write Free software, at the same time. I have a proposal for how to do so: let users pledge contributions toward bugfixes and feature implementations. The basic idea is that the issue trackers used by many Free software projects could easily be extended to allow users to pledge contributions toward particular programming projects; when those projects are completed, each user would pay their pledge to the programmer or team who submitted the patch.
Imagine the following scenario, for example. Janet notices a bug in the way her wireless card works in Ubuntu. She searches on Launchpad and discovers that others have the same problem; there is an open bug report for her issue. Though she has no programming knowledge herself, she is willing to pay $50 toward getting this bug fixed, because it is a major inconvenience and she has no workaround. She clicks a “Pledge now” button in the Web interface and enters this amount. Meanwhile, Katherine, a savvy kernel hacker, notices that a lot of people are having the same problem, and spends some time improving the wireless driver for Janet’s card. Her patch fixes the issue, and the bug is closed. Janet (and anyone else who pledged) are notified that a patch is available, and that they must forward their payments to Katherine. Katherine earns a modest sum for her work, allowing her to spend more time fixing wireless card issues than she otherwise would be able to.
Obviously, there are a lot of logistical problems to be solved here, such as:
- What happens when multiple patches are required to fix a bug, or multiple programmers work on the fix? How should the payment be distributed among patch writers, package maintainers, and others who contribute to improving the quality of the software?
- How do users get a hold of the new version? Do they wait for the patch to move through the normal packaging process before sending their payments? (What if a fix is already available upstream at the time a pledge is made?)
- Who collects the payments, and who ensures that they are made? (Would an arbitration system be needed?)
- What happens to pledges if a bug is marked invalid, or made a duplicate of another?
- What if the patch fixes the problem for some users, but not others?
I think these problems can be addressed by a little engineering and/or “social” innovations. They make a case for experimenting with different informal payment mechanisms to see what works best; they don’t constitute an argument against pledge-based payment systems in general.
The real hurdles presented by this approach are more abstract. They crop up whenever money is introduced where it was previously absent. Payment systems could significantly re-structure Free software communities, and possibly even threaten their existence. Although a payment system could help users exercise their freedom and help programmers to earn a living writing Free code, it might do so at the expense of the volunteerism and collaborative spirit that have made the Free software movement so successful. Programmers who currently work on a project as much as they are able, for their own satisfaction, might become resentful if others start earning money to do the same kind of work. Project leaders might find themselves having to manage payments, in addition to the difficult technical and social problems they already work to solve. And while some users would be able to exercise their freedom more effectively than they can now, the extent to which they can do so would essentially be a matter of their ability to pay, a difficult moral problem in its own right.
For these reasons, we should proceed with caution. I think organizations like Canonical can and should help design these payment systems; after all, they have an interest in making money by developing and distributing Free software. But it should be up to the teams behind individual projects to participate in any payment system, and to decide how they will distribute payments among their members.
Someone once asked me in an interview: “What is your favorite Unix tool?” Not sure of what to say, I simply tried to avoid appearing ignorant, and replied: “grep.”
Thinking about this problem again, I realized that there is a much better answer. Today, my favorite Unix tool is: the pipe.
I have struggled, off and on, to get Gnus, the Emacs mail reader, set up to read my email. Since I’m mostly coming from the world of GUI email clients, without any experience using a newsreader (not that I wouldn’t like to — I’ve just never had access to a news server), Gnus presented some conceptual hurdles.
Gnus treats mail like news, meaning you “subscribe” to various mailboxes, and once read, email is simply hidden unless you explicitly ask to see it. The idea is that the activities of reading mail and news are very similar — you want to see what’s new without having the old stuff around, sort or filter incoming messages, write responses, then get on with your day — so the protocol used to access them (NNTP vs. POP, for example) shouldn’t make a difference in how you do them. Gnus therefore unifies these activities in a common interface, and deals with different protocols and storage methods with different “virtual servers,” i.e., back-ends which find mail or news wherever they may be. To use Gnus, you must learn how to tell it the details of how to access your messages via one or more back-ends.
So, I went to try to find the right back-end for an IMAP mail account. This raises a further complication: unlike most of the mail back-ends, which assume mail is stored locally, IMAP stores messages on the server. This makes the IMAP back-end more like a news server back-end than a mail spool back-end.
After reading the manual section on IMAP, I had tried a variety of settings that didn’t quite seem to work: although I could connect to the server, and read my mail from the Server Buffer, I couldn’t seem to “subscribe” to any “groups”, or split my IMAP mail into these groups. Fortunately, this excellent article provided a complete, working IMAP example. I’m hoping to be a Gnus addict by the end of the summer.
UPDATE: I’ve been successfully using Gnus for a few months now. Here are some relevant bits from my .gnus file to get any new users started:
;; GMANE is about the only free news server I've seen. ;; I set it to my primary server so I can read a few Free software mailing lists. (setq gnus-select-method '(nntp "news.gmane.org")) ;; Mostly, though, I just want to read my mail. ;; This setup uses a standard SSL-based connection to read the mail for the accounts I have through ;; UC Berkeley: (setq gnus-secondary-select-methods '((nnimap "calmail" ; primary email (nnimap-address "calmail.berkeley.edu") (nnimap-server-port 993) (nnimap-authenticator login) (nnimap-expunge-on-close 'never) (nnimap-stream ssl)) (nnimap "ocf" ; secondary account (nnimap-address "mail.ocf.berkeley.edu") (nnimap-server-port 993) (nnimap-authenticator login) (nnimap-expunge-on-close 'never) (nnimap-stream ssl))))
One of the most frustrating things I do in my daily work is wrangle with CSS. Trying to get a particular website to display correctly on multiple browsers and screen resolutions without ruining my HTML semantics can be an extremely painful process. Anyone who’s ever had to do this knows the reason: browsers implement CSS differently, and sometimes (in the case of Internet Explorer) incorrectly and without any sort of sanity. The causes behind these different implementations are various, but they include varying interpretations of the standard, disregard for the standard, and the fact that users cannot submit patches to improve CSS support in proprietary browsers.
Groaning about CSS bugs in Internet Explorer is a favorite topic on forums across the Web, but most people seem resigned to their existence and (begrudgingly) implement their own hacks and fixes, including reverting to table-based layouts. I have never seen or heard a cogent discussion of the reasons why Internet Explorer is the way that it is, or what can be done about it.
This problem is not unique to web browsers and web design. It is an inherent tension present whenever proprietary software is used to read and write data that’s supposed to adhere to an open standard. Open standards tends to evolve; proprietary software does not tend to evolve to meet them. This is partly due, I assume, to a resource allocation problem: it’s possible that even huge proprietary software companies like Microsoft can lack the resources (or willpower) to stay abreast of changes in standards over which they have little control. A bigger contributor is psychology: a culture of software ownership does not tend to foster the values of deferring to standards for the good of a group, sharing code and patches, or contributing in any other way to a community which does not recognize software property “rights.”
I have a second-generation MacBook (bought in spring of 2007), and due to a bunch of recent issues which have turned me off from OS X (e.g., the machine randomly will fail to wake up from sleep, compiling software is a huge pain, etc.), I have been aroused from my complacency and have decided to try my hand at getting gNewSense, an all-free software derivative of Ubuntu, running. I didn’t expect this to be an entirely painless or automatic process — I had tried Ubuntu on the same hardware a little over a year ago, and found it to be lacking — but I needed a distraction from studying for the GRE, and so I dove in last weekend and burned a LiveCD.