NumPy another Iverson Ghost

Posted on by
1

During my recent SmugMug API and Python adventures I was haunted by an Iverson ghost: NumPy

An Iverson ghost is an embedding of APL like array programming features in nonAPL languages and tools.

You would be surprised at how often Iverson ghosts appear. Whenever programmers are challenged with processing large numeric arrays they rediscover bits of APL. Often they’re unaware of the rich heritage of array processing languages but in NumPy's case, they indirectly acknowledged the debt. In Numerical Python the authors wrote:

“The languages which were used to guide the development of NumPy include the infamous APL family of languages, Basis, MATLAB, FORTRAN, S and S+, and others.”

I consider “infamous” an upgrade from “a mistake carried through to perfection.”

Not only do developers frequently conjure up Iverson ghosts. They also invariably turn into little apostles of array programming that won’t shut up about how cutting down on all those goddamn loops clarifies and simplifies algorithms. How learning to think about operating on entire arrays, versus one dinky number at a time, frees the mind. Why it’s almost as if array programming is a tool of thought.

Where have I heard this before?

Ahh, I’ve got it, when I first encountered APL almost fifty years ago.

Yes, I am an old programmer, a fossil, a living relic. My brain is a putrid pool of punky programming languages. Python is just the latest in a longish line of languages. Some people collect stamps. I collect programming languages. And, just like stamp collectors have favorite stamps, I find some programming languages more attractive than others. For example, I recognize the undeniable utility of C/C++, for many tasks they are the only serious options, yet as useful and pervasive as C/C++ are they have never tickled my fancy. The notation is ugly! Yeah, I said it; suck on it C people. Similarly, the world’s most commonly used programming language JavaScript is equally ugly. Again, JavaScript is so damn useful that programmers put up with its many warts. Some have even made a few bucks writing books about its meager good parts.

I have similar inflammatory opinions about other widely used languages. The one that is making me miserable now is SQL, particularly Microsoft’s variant T-SQL. On purely aesthetic grounds I find well-formed SQL queries less appalling than your average C pointer fest. Core SQL is fairly elegant but the macro programming features that have grown up around it are depraved. I feel dirty when forced to use them which is just about every other day.

At the end of my programming day, I want to look on something that is beautiful. I don’t particularly care about how useful a chunk of code is or how much money it might make, or what silly little business problem it solves. If the damn code is ugly I don’t want to see it.

People keep rediscovering array programming, best described in Ken Iverson’s 1962 book A Programming Language, for two basic reasons:

  1. It’s an efficient way to handle an important class of problems.
  2. It’s a step away from the ugly and back towards the beautiful.

Both of these reasons manifest in NumPy‘s resounding success in the Python world.

As usual, efficiency led the way. The authors of Numerical Python note:

Why are these extensions needed? The core reason is a very prosaic one, and that is that manipulating a set of a million numbers in Python with the standard data structures such as lists, tuples or classes is much too slow and uses too much space.

Faced with a “does not compute” situation you can either try something else or fix what you have. The Python people fixed Python with NumPy. Pythonistas reluctantly embraced NumPy but quickly went apostolic! Now books like Elegant SciPy and the entire SciPy toolset that been built on NumPy take it for granted.

Is there anything in NumPy for programmers that have been drinking the array processing Kool-Aid for decades? The answer is yes! J programmers, in particular, are in for a treat with the new Python3 addon that’s been released with the latest J 8.07 beta. This addon directly supports NumPy arrays making it easy to swap data in and out of the J/Python environments. It’s one of those best of both worlds things.

The following NumPy examples are from the SciPy.org NumPy quick start tutorial. For each NumPy statement, I have provided a J equivalent. J is a descendant of APL. It was largely designed by the same man: Ken Iverson. A scumbag lawyer or greedy patent troll might consider suing NumPy‘s creators after looking at these examples. APL’s influence is obvious. Fortunately, Ken Iverson was more interested in promoting good ideas that profiting from them. I suspect he would be flattered that APL has mutated and colonized strange new worlds and I think even zealous Pythonistas will agree that Python is a delightfully strange world.

Some Numpy and J examples

Selected Examples from https://docs.scipy.org/doc/numpy-dev/user/quickstart.html Output has been suppressed here. For a more detailed look at these examples browse the Jupyter notebook:  NumPy and J Make Sweet Array Love.

Creating simple arrays

 
 # numpy
 a = np.arange(15).reshape(3, 5)
 
 NB. J
 a =. 3 5 $ i. 15

 # numpy
 a = np.array([2,3,4])
 
 NB. J
 a =. 2 3 4
 
 # numpy
 b = np.array([(1.5,2,3), (4,5,6)])
 
 NB. J
 b =. 1.5 2 3 ,: 4 5 6

 # numpy
 c = np.array( [ [1,2], [3,4] ], dtype=complex )
 
 NB. J
 j. 1 2 ,: 3 4
 
 # numpy
 np.zeros( (3,4) )
 
 NB. J
 3 4 $ 0
 
 # numpy - allocates array with whatever is in memory
 np.empty( (2,3) )
 
 NB. J - uses fill - safer but slower than numpy's trust memory method
 2 3 $ 0.0001

Basic operations

 
 # numpy
 a = np.array( [20,30,40,50] )
 b = np.arange( 4 )
 c = a - b
 
 NB. J
 a =. 20 30 40 50
 b =. i. 4
 c =. a - b
 
 # numpy - uses previously defined (b)
 b ** 2
 
 NB. J
 b ^ 2

 # numpy - uses previously defined (a)
 10 * np.sin(a)
 
 NB. J
 10 * 1 o. a
 
 # numpy - booleans are True and False
 a < 35
 
 NB. J - booleans are 1 and 0
 a < 35

Array processing

 
 # numpy
 a = np.array( [[1,1], [0,1]] )
 b = np.array( [[2,0], [3,4]] )
 # elementwise product
 a * b

 NB. J
 a =. 1 1 ,: 0 1
 b =. 2 0 ,: 3 4
 a * b

 # numpy - matrix product
 np.dot(a, b)

 NB. J - matrix product
 a +/ . * b  
 
 # numpy - uniform pseudo random
 a = np.random.random( (2,3) )
 
 NB. J - uniform pseudo random
 a =. ? 2 3 $ 0
 
 # numpy - sum all array elements - implicit ravel
 a.sum(a)
 
 NB. J - sum all array elements - explicit ravel
 +/ , a
 
 # numpy
 b = np.arange(12).reshape(3,4)
 # sum of each column
 b.sum(axis=0)
 # min of each row
 b.min(axis=1)
 # cumulative sum along each row
 b.cumsum(axis=1)
 # transpose
 b.T     

 NB. J 
 b =. 3 4 $ i. 12
 NB. sum of each column
 +/ b
 NB. min of each row
 <./"1 b
 NB. cumulative sum along each row
 +/\"0 1 b
 NB. transpose
 |: b

Indexing and slicing

 
 # numpy 
 a = np.arange(10) ** 3 
 a[2]
 a[2:5]
 a[ : :-1]   # reversal

 NB. J
 a =. (i. 10) ^ 3
 2 { a
 (2 + i. 3) { a
 |. a

Setting SmugMug Print Size Keywords with Jupyter and Python

Posted on by
Reply

For the last few weeks, my normal hobbies have been preempted by my obsessive refactoring of an old SmugMug keyword setting program. Many years ago I wrote a small application that helped me maintain meaningful SmugMug keywords. I used this application for years until inevitable software changes broke it. Replacing it with something better has been on my to-do list for ages.

I kept putting it off because I knew the tools I used in the past, mostly J and VB, were not well suited to REST API programming. Yes, I know just about any programming tool can be forced to serve, Turning completeness guarantees it, but I have grown weary of programming to prove tiresome points. If a tool makes something more difficult or tedious than it should be then change tools!

The downside of changing tools is that it usually results in learning yet another programming language. I always try to avoid this! I briefly considered languages with decent REST support that I already sort-of-know1, C#, JavaScript, VB.Net and decided that proper 64 bit versions were either too expensive (C#, VB.Net) or that I didn’t want to stare at ugly bracket infested syntax for weeks. I’m talking about you JavaScript. Even your fanboys admit that you’re an ugly duckling.

I eventually decided to go with Python. I’ve always found Python code more readable than many other programming languages. This is a widespread opinion. Python is free, open source, and comes with a large diverse set of tools. One of the best tools to come out of the Python world is Jupyter. Jupyter is the first public domain open source literate programming tool that has gained a large following.

I’ve been a fan of literate programming ever since I read Knuth’s seminal book. He used the technique to create some of the best program documentation ever created. I always wondered why literate programming never caught on. I suspected the basic problem was simply that many programmers are not particularly literate. Well, Jupyter is proving me wrong.

Jupyter is certainly helping me write and program with such clarity that all my ideas seem trivial.

Compare this notebook, (use the first link for the best layout), to my earlier blog post about setting SmugMug print size keywords.

  1. Nbviewer: Setting SmugMug Print Size and Geotag Keywords with Jupyter and Python
  2. GitHub: Setting SmugMug Print Size and Geotag Keywords with Jupyter and Python
  1. Be highly suspicious of people who claim to fully understand any programming language. Only delusional nitwits would make such a claim for any natural language. Does anybody, even luminaries like Shakespeare, truly understand English? We all sort-of-know our mother tongues and if we’re honest, we’re continually surprised by how others make use of it. Literature, it’s a thing. The same holds for programming languages. Every day I’m surprised, by unusual, stupid, silly, clever and freaking brilliant code fragments in programming languages that I have used for decades.↩

Government Shutdown and Rapture 2018

Posted on by
Reply

It’s 2018. 2018 has prime factors of 2 and 1009, e.g. 2018 = 2 * 1009.

Did anything happen in the years 2 and 1009?

  1. In the year 2 Venus and Jupiter were in conjunction. Some speculate this may have been the “Star of Bethlehem.”
  2. In 1009 the Church of Holy Sepulcher was destroyed.

How odd, the prime factors of 2018 yield two Jesus related events, separated by over a thousand years that are mysteriously tied to his birthplace. Evidently, the Star of Bethlehem marked the beginning, 1009 marked the middle, and 2018 is the end! Repent, the rapture is upon us! Better buckle up non-gender specific bovine control officers (cowboys) 2018 is going to be rough.

Just how rough you ask? Imagine hordes of underachieving and mostly useless bureaucrats forced to take an impromptu paid holiday1 because even more useless legislators cannot agree on how to spend other people’s money. Let’s hope the government stays shutdown down until the imminent rapture renders it redundant.

My calculations cannot pinpoint the exact time and date of the rapture. My guess would be the first2 blood moon of the year, January 31, 2018. Get your affairs in order and divest yourself of your worldly bitcoins.3 Use the address encoded in the QR graphic of this blog.

Hey: If you think this micro-epiphany is batshit crazy take a look at this! It’s getting to the point where it’s no longer possible to be satirical because somewhere on the intertubes you’ll find true believers that go way beyond satire.

  1. None of the furiously furloughed will miss a single damn paycheck. Where can I get a job that pays me to sit on my ass and whine about whatever the idiot talking point of the day is?↩
  2. 2018 has two total lunar – blood moon – eclipses. If we’re not raptured on January 31st wait until July 27th↩
  3. Sky Fairy consultants, also known as prophets, have high overhead and need substantial donations to maintain their close connections with the divine.↩

Dear Apple Pay Stop Harassing Me!

Posted on by
Reply

Dear Apple Pay stop harassing me! I will never sign up for you and no amount of iOS app-nagging will change that. You’re pathetic interruptions merely remind me why I “resist” installing every freaking iOS upgrade the Apple mothership foists on its hapless phone users.

I use an iPhone1 but I am not a member of the bleating iSheep. I’ve slowly, one unwanted upgrade after another, developed an intense loathing of Apple’s holier than thou, we know what’s best for the iIdiots, attitude. You’re beginning to remind me of the Hildabeast. Remember that foul creature and her well-deserved fate!

Loathing aside, I have a perfectly logical reason for refusing Apple Pay. Unlike the bleating iSheep, I see what you are trying to do. You want to become a financial “middleman.” You want to skim an ever-increasing percentage of every transaction your well-trained herd of iSheep makes. I admire your larcenous spirit but if you haven’t noticed, we have enough parasitic financial rent-seeking middlemen on this insane planet. Part of the allure and promise of cryptocurrencies is that they show a way to rid the universe of skimming scum like central banks, government fiat, rapacious money transmitters, exchange controls and abominations like Apple Pay.

Don’t ever darken my day with another “activate Apple Pay now” message again. You have been warned!

  1. Unless there are big changes at Apple this is my last iPhone. I want a device that I absolutely control. “If you don’t control it you cannot trust it.” Any phone that I cannot even be sure is off does not qualify. Burners may be my only option.↩

Downloading SmugMug Captions with Python and Jupyter

Posted on by
Reply

This blog post started out as an experimental rendering of a Jupyter notebook. I wanted to see how difficult it would be to convert a notebook to a WordPress.com blog post. Even though Jupyter exports notebooks in HTML and Markdown they do not display well “out of the box.” No doubt one could craft CSS that would help but the entire point of Jupyter exports is to cut down on pointless format thrashing.

This post is a teaser. If you want to get to the source notebook follow this link to my GitHub repositories.

If you only want to read the notebook use this nbviewer link.

Why am I doing this?

My photo captions have evolved into a form of milliblogging. Milliposts (milliblog posts) are terse and tiny; many are single sentences or paragraphs. Taken one-at-a-time milliposts seldom impress but when gathered in hundreds or thousands accidental epics emerge. So, to prevent “epic loss” I want a simple way of downloading and archiving my captions off-line.

If you don’t control it you cannot trust it!

When I started blogging I knew that you could not depend on blogging websites to archive and preserve your documents. We had already seen cases of websites mangling content, shutting down without warning, and even worse, censoring bloggers. It was a classic case of, “If you don’t control it you cannot trust it.” I resolved to keep complete off-line version controlled copies of my blog posts.

Maintaining off-line copies was made easier by WordPress.com‘s excellent blog export utility. A simple button push downloads a large XML file that has all your blog posts with embedded references to images and other inclusions. XML is not my preferred archive format. I am a huge fan of LaTeX and Markdown: two text formats that are directly supported in Jupyter Notebooks. I wrote a little system that parses the WordPress XML file and generates LaTeX and Markdown files. Yet, despite milliblogging long before blogging, I don’t have a similar system for downloading and archiving SmugMug metadata. This Jupyter notebook addresses this omission and shows how you can use Python and the SmugMug API to extract gallery and image metadata and store it in version controlled local directories as CSV files.

Review: The Way We Die Now

Posted on by
Reply

The Way We Die Now is not the best book I’ve read this year but it may be the most important. In Seamus O’Mahony’s opinion, modern society has forgotten how to deal with death. There are many reasons for this, the collapse of religious belief, the demolition of the extended family, the triumph of the scientific and rational worldview, even our delusions of curing death “real soon now” contribute to our collective denial. Yet death persists. Death remains absolute, sovereign, implacable, terrifying, “majestic and cruel.” Even if we realize our singularity fantasies and greatly extend life death will never be banished. Even the gods die! We must face death, but must we turn it into a carnival of “medical excess?”

I have seen medical excess. My mother was diagnosed with Stage IV Glioblastoma: a form of brain cancer that is so deadly it’s been nicknamed the terminator. Actually, the terminator is flattered by the comparison. Some survived their encounter with fictional terminators. Nobody survives stage IV Glioblastoma: “there is no stage V.” When I heard mom’s diagnosis I looked for actuarial survival statistics. Credible statistics for common fatal diseases are harder to track down than you might expect. I eventually found a paper that cast survival times in a useful form. Median survival was less than three months for younger and healthier patients than my mother. She died about two months after her diagnosis – right on statistical schedule. The universe does not make personal exemptions.

Her death was inevitable, but the expensive, futile, painful and isolating medical gauntlet she endured was not. She just wanted to go home, perhaps to “turn her head to the wall,” perhaps to binge on The Big Bang Theory – she still enjoyed a few silly shows. It doesn’t matter what the dying choose to do with their remaining hours, but it sure as hell matters that we honor their choices and the Way We Die Now makes a compelling case that we are failing “to be brave.” I know I acquiesced to the medial default for my mother; I still feel I should have fought harder for what she wanted.

According to O’Mahony, the medical default is full intervention even when it’s pointless and wasteful. He also notes that doctors are in a no-win situation. If they suggest doing nothing they’re accused of euthanizing patients. If they go full interventionist Rambo they’re inflicting needless suffering and profiting from the dying. Both extremes often end up in court, as if we could fix death with more litigation. Obviously, something in the middle is the best course and O’Mahony argues that doctors should not set the middle course.

Our infantile society needs to grow up and face death like adults. Nothing makes our magical thinking about death clearer than Somerset Maugham’s1 observations about a “dog’s death.” Maugham hoped he was lucky enough to die a dog’s death! A dog’s death is meant to be a horrible thing but is it really worse than human medical excess? When it comes to sick animals we are clear-headed and compassionate. We don’t subject them to futile treatments, we make them comfortable and take away their pain. I once had a cat that came down with pancreatitis. She wasted away on the top of our fridge until one day we took her to the vet. Her death was calm and without terror. My cat had a better death than my mother. I suspect many pets die with greater dignity than their owners. This is fundamentally wrong and we all know it.

There are no easy answers; it sucks to be mortal. We can’t say until we face it ourselves how we should die so how can we dictate to others? I only hope that when my time comes I have it within me to follow the one bit of advice O’Mahony offers that may apply to all us – “be brave.’

  1. The Way We Die Now relates many stories about “celebrity” deaths.↩

The Mass Kill as Performance Art

Posted on by
1

It’s been almost a week since the Las Vegas Mandalay Bay massacre and the idiot media is still looking for “a motive.” They remind me of O.J.’s fruitless search for the “real killer.” I don’t watch the alphabets, except when trapped in airports or, with increasing annoyance, in my employer’s cafeteria so I’ve missed days of mindless speculation but my limited TV sampling confirms what’s easily gleaned from more efficient news sources. The killers “motive” is still unknown and the authorities are still looking.

In all the blather about the mysterious highly organized and thoughtful killer, it’s never occurred to anyone (on TV anyway), that the lack of an apparent motive is exactly what the killer hoped to achieve.

Consider the usual mass media slaughter script. An individual, or group of individuals, attack and murder a sufficiently1 large number of “innocents.”

If, as is often the case, the attackers are well-known terrorists they will typically gloat and issue more threats. Public threats trigger the idiot media’s “analysis.”

If the terrorists are Jihadis the idiot media will downplay the attack while issuing stern warnings about not jumping to racist conclusions about an “entire religion.” Then, when forced by competing news organizations like Fox or right-leaning bloggers, they will join the fray and condemn the killers while searching for a way to blame Trump.

If the killer is black and the victims are black — well this isn’t news! Press the ignore button and complain about anything that can be plausibly blamed on Trump. And, fortunately for the idiot media, that well will never run dry.

If the killer is white and the victims are mostly white (bingo for Las Vegas), spend a day or two glorifying the murderer. Review his2 typically pathetic and meaningless life while running candlelight vigils, peace garden plantings, and out-of-tune Kumbaya-a-thons in the background. In more sober moments touch on the “known motive” for the mayhem. Tut-tut the violence, reassure moronic viewers that violence is never the answer. Play a few bars of Imagine then, when appropriate mourning turns to anger, use the “known motive” to pivot to what the idiot media wanted to talk about from the beginning: gun confiscation.3

The sooner the idiot media can get “the motive” out of the way the sooner they can get down to one of their favorite topics. Sadly, the Vegas killer, being an intelligent and detail oriented psychopath, anticipated this and left no clear motive forcing the idiot media to fixate, like a dog licking its ass, on “the motive” for day, after day, after day. Every day spent discussing “the motive” inflates the killer’s infamy. He’s already the most famous mass killer in recent history.

Real villainy requires incomprehensible dread and clear motives eliminate dread. When Jihadis kill it’s no big deal. Their sky fairy manual exhorts true believers to behead, enslave and tax infidels. It’s hard to imagine a violence-free way to realize such goals. When black hating psychopaths open fire in churches the idiot media have ready-made dread dispelling explanations. If you can quickly explain an atrocity it almost ceases being atrocious. But, if the horror can never be explained, if it sticks in your existential craw, it remains a source of terror forever. This is what the killer was really aiming for.

I view the Las Vegas massacre as deadly performance art. The killer has more in common with “artists” that drop their pants in public and pee on crucifixes than cause-driven revolutionaries or garden variety psychopaths. What exactly motivates public “performance” crucifix urination? To even ask the question is to mock it. For one deadly night the Vegas killer staged a performance that upstaged all the other Vegas shows and unlike another showing of Cirque du Soleil or Menopause the Musical his performance will be remembered forever. The mass kill is a new art form and the idiot media is its biggest patron.

  1. Sufficiently large is an ever-increasing natural number that is now greater than ten.↩
  2. The overwhelming majority of mass killers are male.↩
  3. It’s impossible to have an intelligent debate about guns in the US is because neither side is willing to discuss their ultimate goals or take responsibility for their positions. Gun controllers secretly want the second amendment appealed and all existing guns confiscated. Nothing else will satisfy them. They naively think an unarmed society will never be abused by the state. Gun holders often point out that the second amendment was never about creating a safe and secure society but about erecting deadly barriers to government tyranny. Unfortunately, they’re unwilling to admit that a heavily armed populace will result in large numbers of firearm deaths. US gun deaths are an order of magnitude greater than comparable western societies: that tree of liberty is more bloodthirsty than expected.↩