Blog Archive: August 2016
More photo processing stuff
Lots more image scans today, in the process improving my EXIF creation script. I've let myself in for a lot of work. ACM only downloads articles once. It's possible that this article has changed since being downloaded, but the only way you can find out is by looking at the original article.
USB cables should be "charge-only" by default
Every sysadmin knows that you can protect a server though cryptographic or other means, but if someone has physical access "all bets are off". Right? With physical access they can do physical damage (smash it with a hammer) or pull out a hard disk and read the bits directly. Even security systems that are highly respected (I'm looking at you, Kerberos!) are an "all bets are off"-situation if someone gets the private key through physical access. Sadly we forget this when it comes to smartphones. We'll plug our phones into any darn USB charger we find... especially when we are desperate!
Factors: enough!
A month ago I started doing some factorizations of large numbers. The last number I tried was 1235443534534543534435425454254254554254325542543254325524524543254325425243543, and it kept the machine busy for quite some time. In fact, today, after a month, it was still going strong: USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND grog 2282 100.0 0.0 16484 3680 38 R+ 27Jul16 44835:12.87 /usr/games/factor 123544353... That's 44835 minutes of CPU time (1868 hours, or 31 days). That's enough, and I pulled the plug.
Getting to the bottom of scanning
Did some more playing with the crop rectangles on the Home mode previews, and was able to move them around or diminish them, but not to enlarge them: Reading the reference guide, however, was illuminating. It seems that there is a way to remove the crop: just click on the Delete Marquee button. And sure enough, it's gone, and I can scan the entire area of the slide (which it recognizes correctly). Only: this only works in toy Home mode, which of course is far too simple to understand 16 ( 48 ) bit pixel depths.
Still more photo processing
Gradually I'm coming to terms with scanning the old photos, and about the only issue I still have is the excessive cropping, which I haven't been able to fix. Found a reference guide for both the scanner and the software, conveniently in multi-page, hard to navigate HTML, but at least it contains some information. Not, unfortunately, information on how to change the crop. About the only thing I discovered is that while the Home settings display the crop (and confirm that it's too small), I can't change it: It did, however, tell me that some of the buttons in the Adjustments section create additional windows that modify the function.
See you at Burning Man!
I’m about to switch off my email until September 5 and drive to Black Rock City for 10 days of incinerating the dude. If you’re going this year, drop by Liminal Labs — with whom I am immensely privileged to camp — and have some cold brew and say hi! We’re at 5&F this year... more
EXIF creation script: done?
More playing around with my script for entering EXIF data into the scanned photos today, and got something relatively useful. Clearly there's material for much more playing around here. ACM only downloads articles once. It's possible that this article has changed since being downloaded, but the only way you can find out is by looking at the original article.
Still more scanning and processing
Why does my scanner no longer show thumbnails in professional mode? If it's a setting, maybe the old Microsoft reinstall trick will work. And maybe it won't. And maybe I'll just lose all scanner functionality. A safer bet would be to try it on a second machine. Did a bit of digging and found the disc for dxo.lemis.com, the predecessor of dischord, running Microsoft Windows Vista. And how about that, it worked normally. So it seems most likely that the problem was some interference from SilverFast. I'll leave it the way it is for the time being, until I decide whether I want to use SilverFast or not.
Talking about the pro-security, anti-DRM business model on the OReilly Radar Podcast
On this just-released episode of the O’Reilly Radar podcast (MP3), I talk about EFF’s lawsuit against the US government to invalidate Section 1201 of the DMCA, which will make it legal to break DRM in order to fix security vulnerabilities in the Internet of Things devices that, today, are almost invariable insecure, and are also... more
Automating a CI environment for Android Apps
One of the things my team at StackOverflow does is maintain the CI/CD system which builds all the software we use and produce. This includes the Stack Exchange Android App. Automating the CI/CD workflow for Android apps is a PITA. The process is full of trips and traps. Here are some notes I made recently. First, [this is the paragraph where I explain why CI/CD is important. But I'm not going to write it because you should know how important it is already. Plus, Google definitely knows already. That is why the need to write this blog post is so frustrating.]
The Fixing-JSON Conversation
Last week I suggested some modest JSON improvements, and conversation ensued. Obviously, much was Hes Wrong On The Internet (again) but lots was juicy and tasty, and worth considering further. This is based on reaction in my own comments, and on Hacker News. Just use X For values of X including Hjson, Amazon Ion, edn, Transit, YAML, and TOML. Nah, most of them are way, way richer than JSON, often with fully-worked-out type systems and Conceptual Tutorials and so on. I just want JSON thats easier to edit and can do timestamps. Hmm, SON looks pretty close to what Im thinking.
Breakfasts of the world
Quora is a strange site. People ask questions, anybody can answer them, including myself. But many questions are just plain silly, and there are very qualified people who answer many of them, so I seldom bother. Recently there was a question on the Zika virus. A good answer came from a certain Hillary Clinton (slamming the Republican party for not doing more). And another asked what kind of vice-president Tim Kaine would make. Answer came from Tim Kaine. So clearly it's not completely irrelevant. But by coincidence I found, a while back, a breakfasts of the world post. There's some interesting stuff in there, and of course some stuff that I'd never think of touching (typically overly sweet and processed US food), but a surprising number of suggestions of the kind I've been looking for for some time.
Goodbye Android
My Android tablet is not even 3 years old, but it has become almost impossible to charge it: both power inputs make such poor contact that it's hard to tell whether it's charging or not, not in the slightest helped by the fact that the device takes about 20 seconds to show any reaction when a charger is connected. Over the past few days it seems to be suffering from a further issue, possibly dead battery: while connected to the charger it would intermittently display the logo (the first thing it does when starting to charge), and then turn off again. Today I moved it from my desk to my battery charge area on the other wall, and tried to turn it on.
Fixing JSON
Ive edited a couple of the JSON RFCs, and am working on the design of a fairly complex DSL, so I think I can claim to have dug deeper in the JSON mines than most. We can easily agree on whats wrong with JSON, and I cant help wondering if itd be worth fixing it. Major irritant: Commas Hand-editing JSON may not be the most important way of interacting with it, but it shouldnt be as hard as it is. In particular, when Im moving things around in a chunk of JSON I can never, as in NEVER, get the commas right.
Podcast: Live from HOPE on Radio Statler
While I was in NYC to keynote the 11th Hackers on Planet Earth convention, I sat down with the Radio Statler folks and explained what I was going to talk about, as well as bantering with the hosts about the relative merits of DEFCON and HOPE and the secret to managing cons and marriages (MP3).
Failed panorama
I had to take photos of yesterday's picture hanging, of course. The first idea was a panorama taken from the middle of the (circular) table in the middle of the room. How do you mount the camera? It has to be placed on top of the table, of course. But with the panorama bracket my real tripod is about 85 cm high, probably too high for the view I wanted. So I mounted it on a toy tripod I bought years ago. There's a carousel ( lazy Susan ) in the middle of the table, so I used that as a rotator. The results were underwhelming: Yes, of course I can crop the table surface, and I did: ...
Podcast: How well kill all the DRM in the world, forever
I’m keynoting the O’Reilly Security Conference in New York in Oct/Nov, so I stopped by the O’Reilly Security Podcast (MP3) to explain EFF’s Apollo 1201 project, which aims to kill all the DRM in the world within a decade. A couple things changed in the last decade. The first is that the kinds of technologies... more
File upload: no go
My file upload web page works, sort of, at least for me. Today we tried it in earnest, and one limitation became evident immediately: only one file at a time. Possibly there's a solution for that, but there was worse to come: it just failed silently. Select a file, press submit, and no response, not even the debug output that should have appeared. How do you debug that? One thing is clear: not during the presentation. Paul suggested using Dropbox instead, which sounded like complete overkill to move relatively large quantities of data 10 metres across the local network. But he signed me up anyway, and we waited for a confirmation message.
Camera Crew meeting
The monthly Dereel Camera Crew meeting took place this afternoon at my place, where I presented my raw processing presentation. It's not a big group, but I was expecting more than the two people who came (Carol Ann Moyse and Paul Shire). Paul, a professional photographer, was particularly interested, as he should be, and had lots of questions. How well did I address things? I'm not sure. It seems that the illustrations of how pixels work really was worthwhile. as were the underexposure examples. But Paul knew how to fix the underexposed image with Photoshop It'll only take me 10 seconds . Five minutes later he came up with an image which really did lighten up the shade area and keep the skies from burning out.
Light Transmission
Lots of photographers, and people who teach them, talk about the advantages of shooting under a cloudy sky, or of having the suns light behind you. Increasingly, Im enjoying pointing my camera straight into the sun. Of course, as above, this can force you into a monochrome presentation (although the dock above is in color). But I remember, during my time in publishing technology and then again in VR, learning about emissive as opposed to reflective color. I dont remember the physics or the engineering, but I like it. Of course, the color/monochrome continuum isnt binary. Lets take three steps into total whiteness overload.
Things About No Mans Sky
Of course, I had to pre-order this one after reading World Without End in The New Yorker. Ive only played a few evenings, havent done anything ambitious, but Ive learned that when youre learning something is a good time to write about it. Its just really cool bringing your ship in for a landing on a new planet. There seems to be a consensus that with this game, spoilers are a virtue; I think I would have found it severely frustrating if Id had to puzzle out all the crap I picked up in a quick scan of Reddit & IGN.
Uploading data via web site
On Saturday the Dereel Camera Crew are coming to learn about raw image processing. I'm asking them to bring some sample images with them. And how do we upload them? Loading photos from cameras using commodity operating systems is such a pain that I don't want to get started. So I'll ask them to bring a laptop and upload the images themselves. And how? CIFS? NFS? FTP? The modern way is to use a web page. How do you do that? Simpler than it seems, apparently. This page on W3schools popped up first, but it's clearly based on this php.net page.
More census crash background
It's becoming clearer and clearer that the Census crash had nothing to do with FreeBSD. And it's not becoming much clearer what the real reasons were. But this tweet (a screenshot, of course, because you can't say anything useful on Twitter) seems to offer a plausible description. ACM only downloads articles once. It's possible that this article has changed since being downloaded, but the only way you can find out is by looking at the original article.
More unintelligible error messages
While working on the server overload issue, discovered a number of these messages: pid 63711 (httpd), uid 80, was killed: out of swap space pid 63932 (httpd), uid 80, was killed: out of swap space swap zone ok swap zone exhausted, increase kern.maxswzone I've never seen that before. I wasn't out of swap spaceI was only using about 35%. Took a look and found: === root@www (/dev/pts/0) /usr/local/etc/apache24 89 -> sysctl kern.maxswzone kern.maxswzone: 0 That looks like a default value, the same as I have on my other machines.
Server overload
For some reason I forget, I wanted to look at something on the external server www.lemis.com today. And once again, as a couple of weeks ago, the system was swapping its little heart out. This time things looked different, though: there were up to 100 httpd processes running, apparently all doing something useful. But this machine only has 1 GB of memory, and the swapping just completely overloaded it. OK, there are parameters to limit the number of concurrent servers. What are they again? Once I had a good understanding of Apache, but that was decades ago. Looking in http.conf, found: # Limit on total number of servers running, ...
Census web site down, but no data corruption
Yesterday was census day in Australia, and it seems the majority of the population chose to perform it on line. And then came the catastrophe: Hackers ! They had to close down the site, and as of a day later, it's still down. Typical Microsoft vulnerabilities, right? Went off to find details and gloat. But I wasn't prepared for what I saw: FreeBSD? Is that our fault? Sent off a message to the developers to discuss it, and in the meantime tried to find more details.
I'll be speaking at NYC SRE Meetup next week (Wed)
I'll be giving a new talk called "Teaching DevOps to Ops without Devs" at the NYC SRE Tech Talks meetup, Wednesday, August 17, 2016, from 6pm to 8:30pm, at Squarespace's offices in the West Village (8 Clarkson St, 12th Floor). There are 4 speakers during the event. This is a free event. Seating is limited. Info and registration here.
25 years of web sites
Coincidentally with the 71st anniversary of Little Boy, and almost coincidentally with the 50th anniversary of my acquisition of my oldest camera, it seems that today was also the 25th anniversary of the first web site. How time flies! ACM only downloads articles once. It's possible that this article has changed since being downloaded, but the only way you can find out is by looking at the original article.
New radio
The radio that I ordered at the end of June has finally arrivedwasn't posted until 13 July. It must have the distinction of being the only device I have with both instructions and inscriptions only in Chinese. I worked out how to use it as a radio, but there are other functions that will take time to decipher. At least I can use it for the original purpose, to trace faults in the electric fence. ACM only downloads articles once.
My Most Complex Pattern-Welded Sword
HDR fun, yet again
On Tuesday I finally managed to get a better image from HDR Projects 4, and to save the profile. That related to images with extreme dynamic range. How would that work with the relatively simple scenario of my house photos? Started with that profile and played around a while. And how about that, finally something better! Here are the results for last weekend, first the out-of-box HDR Projects results, then those with enblend, and finally today's results: Today's do look the best, though arguably they're still too gaudy.
Full Google Maps!
Since Google Maps has downgraded to the New Look, it has refused to give me Full Google Maps . Why? It's not saying; the link just gives generics, not specifics to my installation. Sometimes I wonder if it's worth it: it seems so much worse than it used to be that I'm wondering if it's not deliberate. Today, however, the unexpected happened. Looking for contact numbers on the PPT web site, I found an embedded map with an incorrect location: OK, follow the link at the bottom, and I got an unexpected message: OK, nothing I'd like better (well, within context).
My Kansas City World Science Fiction Convention schedule
I’m flying into Kansas City for part of Midamericon II, the 74th World Science Fiction Convention, and while there, I’ll be on panels, give a reading, and sit down with fans for a kaffeeklatsch. Here’s my schedule: Thursday: Is Cyberpunk Still a Thing? Thursday 12:00 13:00, 3501H (Kansas City Convention Center) Cyberpunk hit with... more
Factor: a week and counting
Last week I started running /usr/games/factor on a 78 digit number. It got as far as the first four factors (3 19 83 4259) and then got stuck in a loop looking for the next. It's still going a week later: USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND grog 2282 100.0 0.0 16484 3680 38 R+ 27Jul16 10117:06.11 /usr/games/factor 123544353453454353443542545425425455425 Interestingly, the time 10117 minutes corresponds to 7.026 days, almost exactly the time it has been running.
Understanding HDR, yet again
I'm writing a presentation about raw imaging processing for the Dereel Camera Crew. That's not really necessary: a demonstration would be good enough. But I have a desire to express things as well as possible, and as usual it's a lot of work. Where do you start? Not with raw images, that's for sure. First you need to understand the background, and that's where I'm (currently) starting: the presentation has been through several restructures, and I'm probably not done yet. Clearly the one area where raw images beat the hell out of JPEGs is recovering underexposed images. Here's an image which was roughly 4 EV underexposed: And here it is recovered from JPEG and from raw: ...
More focus stacking
Our Schlumbergera is flowering, another pretty flower with enough third dimension to challenge macro lenses. Took a few photos, both with in-camera processing ( focus stacking ) and postprocessing stacking ( focus bracketing ). The results weren't too bad: But there are still issues with the external software, Focus Projects 3 professional , which I used for the first image. Enlarging the image shows considerable bleed: I haven't seen that with the in-camera processing, like the second image.
CJ's software problems
Call from CJ Ellis today: his Google Chrome no longer worked. How do you diagnose that? In the end he came over with the box and I plugged it in here. Boot up: Windows did not shut down properly . OK, boot up and recover, holding a lecture on proper shutdown techniques while I was at it. And then fired up Chrome, which worked as well as could be expected under the circumstances. All a matter of performance and patience, the lack of which was probably the cause of the problem in the first place. When I shut the machine down again, CJ complained that it took such a long time.
New DxO
Another update to DxO Optics Pro today, requiring a reboot to remind me that this is the Microsoft space. After restarting, I got an apparently benign error message: EntityMemberChanged or EntityComplexMemberChanged was called without first calling EntityMemberChanging or EntityComplexMemberChanging on the same change tracker with the same property name. For information about properly reporting changes, see the Entity Framework documentation. Now isn't that an interesting and intuitive message to the end user? Looks like a debug message that broke out. What does it mean? Race condition? I suppose I should send in a bug report and watch them mishandle it.