Bifurcated Rivets: From FB

Great stuff

Bifurcated Rivets: From FB

Could not be bettered

Bifurcated Rivets: From FB

Badalamenti?s Factorization slide rule Interesting

Bifurcated Rivets: From FB

Area Matters

Bifurcated Rivets: From FB

Interesting

MetaFilter: wet noises intensify

Eels eat pizza

Slashdot: Google Criticized Over Its Handling of the End of Google+

Long-time Slashdot reader Lauren Weinstein shares his report on how Google is handling the end of its Google+ service. He's describing it as "a boot to the head: when you know that Google just doesn't care any more" about users "who have become 'inconvenient' to their new business models." We already know about Google's incredible user trust failure in announcing dates for this process. First it was August. Then suddenly it was April. The G+ APIs (which vast numbers of web sites -- including mine -- made the mistake of deeply embedding into their sites), we're told will start "intermittently failing" (whatever that actually means) later this month. It gets much worse though. While Google has tools for users to download their own G+ postings for preservation, they have as far as I know provided nothing to help loyal G+ users maintain their social contacts... As far as Google is concerned, when G+ dies, all of your linkages to your G+ friends are gone forever. You can in theory try to reach out to each one and try to get their email addresses, but private messages on G+ have always been hit or miss... And with only a few months left until Google pulls the plug on G+, I sure as hell wouldn't still be soliciting for new G+ users! Yep -- believe it or not -- Google at this time is STILL soliciting for unsuspecting users to sign up for new G+ accounts, without any apparent warnings that you're signing up for a service that is already officially the walking dead! Perhaps this shows most vividly how Google today seems to just not give a damn about users who aren't in their target demographics of the moment. Or maybe it's just laziness. I'd be more upset about this if I actually used Google+ -- but has Google been unfair to the users who do? "[T]he way in which they've handled the announcements and ongoing process of sunsetting a service much beloved by many Google users has been nothing short of atrocious," Weinstein writes, "and has not shown respect for Google's users overall."

Read more of this story at Slashdot.

MetaFilter: Creating While Clean; 9 sober musicians

In the modern pop-culture tradition, being a musician has often come with a series of default lifestyle expectations, ones of indulgence and recklessness, larger-than-life living, and a diligent pursuit of altered forms of consciousness. Some see these expectations as having played a part in what happened to them, though most ultimately see their decisions and actions as also—if not mainly—a matter of their own psychology and personality and predisposition...What they have in common is that they are all, by their own account, for now, living sober.

"Some delight in a dark humor about their earlier excesses; others talk in a way that suggests that to dwell on these too much, to give such memories too much oxygen, would be to take too lightly something they simply can't risk taking lightly. That it would be foolhardy or perilous to risk returning, even in thought, to a place where for all kinds of reasons they'd rather not linger. A corollary is that some are reluctant in this context to offer much detail about the particular substances that they consumed, or that consumed them, or both.

And while the particulars they spoke of may be specific to each of them, the wider predicaments and decisions and quandaries and insecurities and dilemmas they spoke of are the same ones that confront us all. No matter which choices each one of us elects to make as we hack through the undergrowth into the future, no matter how like or unlike these lives here might seem to our own lives, I would be astonished—and perhaps a little worried, too—to discover anyone who could read the words these interviewees share without finding plenty to relate to or empathize with, and plenty more to think about."

Recent additions: sets 0.0.6.1

Added by athanclark, Sun Jan 20 06:22:32 UTC 2019.

Ducktyped set interface for Haskell containers.

Slashdot: Is California's PG&E The First Climate Change Bankruptcy?

"California's largest power company intends to file for bankruptcy as it faces tens of billions of dollars in potential liability following massive wildfires that devastated parts of the state over the last two years," reports the Washington Post. Calling it "a climate change casualty," one Forbes contributor notes that PG&E's stock has now lost 90% of its mid-October value after a giant November wildfire, adding that "Future investors will look back on these three months as a turning point, and wonder why the effects of climate change on the economic underpinnings to our society were not more widely recognized at the time." Climate scientists may equivocate about the degree to which Global Warming is contributing to these fires until more detailed research is complete, but for an investor who is used to making decisions based on incomplete or ambiguous information, the warning signs are flashing red... there is no doubt in my mind that Global Warming's thumb rests on the scale of PG&E's decision to declare bankruptcy. And the Wall Street Journal is already describing it as "the first climate-change bankruptcy, probably not the last," noting that it was a prolonged drought that "dried out much of the state and decimated forests, dramatically increasing the risk of fire." "This is a fairly new development," said Bruce Usher, a professor at Columbia University's business school who teaches a course on climate and finance. "If you are not already considering extreme weather and other climatic events as one of many risk factors affecting business today, you are not doing your job"... In less than a decade, PG&E, which serves 16 million customers, saw the risk of catastrophic wildfires multiply greatly in its vast service area, which stretches from the Oregon border south to Bakersfield. Weather patterns that had been typical for Southern California -- such as the hot, dry Santa Ana winds that sweep across the region in autumn, stoking fires -- were now appearing hundreds of miles to the north. "The Santa Ana fire condition is now a Northern California fire reality, " said Ken Pimlott, who retired last month as director of the California Department of Forestry and Fire Protection, or Cal Fire. "In a perfect world, we would like to see all [of PG&E's] equipment upgraded, all of the vegetation removed from their lines. But I don't know anybody overnight who is going to catch up." PG&E scrambled to reduce fire risks by shoring up power lines and trimming millions of trees. But the company's equipment kept setting fires -- about 1,550 between mid-2014 through 2017, or more than one a day, according to data it filed with the state. The global business community is recognizing the risks it faces from climate change. This week, a World Economic Forum survey of global business and thought leaders found extreme weather and other climate-related issues as top risks both by likelihood and impact. Other factors besides climate change may also have pushed PG&E towards bankruptcy, according to the article. They're required by California state regulations to provide electrical service to the thousands of people moving into the state's forested areas, yet "an unusual California state law, known as 'inverse condemnation,' made PG&E liable if its equipment started a fire, regardless of whether it was negligent." In declaring bankruptcy, PG&E cited an estimated $30 billion in liabilities -- plus 750 lawsuits from wildfires potentially caused by its power lines.

Read more of this story at Slashdot.

MetaFilter: Grand!

What if the Grand Canyon were inverted?

MetaFilter: Walter Chandoha's cat models [...] must be alert, graceful and beautiful

In the internet age, we are all cat photographers. One study found that in Britain alone, more than 3.8 million photos and clips of cats are shared each day — twice the number of selfies shared (we love our cats more than we love ourselves) and more than twice the number of dog photos shared. Once upon a time, though, there was but One Cat Photographer to Rule Them All. His name was Walter Chandoha. (Hyperallergic)

By the time he died, on Jan. 11, Mr. Chandoha had taken some 90,000 cat photos, nearly all before cats had become viral darlings of social media. He was 98. (NYTimes) (Non-Times link, leaves out some adorable pics)

One of his best-known pictures, "The Mob," was shot outside his studio. It depicts five determined-looking cats walking on his farm in Annandale, as if looking for trouble. "It was about time for dinner, and I called 'Kitty, kitty, kitty,' and all the cats came running," he told CNN in 2016.

When they slowed down, he dropped to his stomach and captured their fleeting tough-cats-in-the-countryside moment.

MetaFilter: Guitarist Phil Dürr, 1965-2019

Phil Dürr, best known as the guitarist for '90s Detroit funk/riff rock band Big Chief and later for blues singer Thornetta Davis, died at 53 on January 11th. This obituary sums up his life and work in loving detail.

"One Born Every Minute" from Big Chief's Mack Avenue Skull Game.
"Cry" from Thornetta Davis' Sunday Morning Music.
From the linked obit:
That Dürr was a phenomenal guitarist — whose formative influences ranged from Daniel Ash to Jimmy Page to Eddie Hazel — was really the least of his gifts. More importantly, he was a cherished friend to basically everyone he ever crossed paths with, that rare man about whom there is nary a bad word to be said. His sweetness, generosity, humility, and good humor (usually self-deprecating) charmed anyone he encountered; I would bet that there are no less than 50 people spread across several continents who considered him their BFF at various points in their lives. He was that kind and priceless of a fellow.

But don't just take my word for it. Consider instead Jon Fine, the former guitarist of the Oberlin, Ohio, band Bitch Magnet, who shared a testimonial on Facebook about what happened in 1990 when that group's gear got ripped off in London after the initial gig of a six-week European tour. Desperate, Fine bought a crappy guitar to muddle through, but then met Dürr in Germany, who loaned him a gorgeous wine-red Gibson Les Paul Standard for the rest of the tour. Dürr "entrusts me," wrote Fine, "a more or less total stranger whose band he sort of knows, with a very nice instrument, and basically says, 'Here. Use it well. Do your thing. Just send it to me in Michigan when you're done with the tour.'"

Anyone acquainted with Dürr hearing this story would immediately reply: "That's so Phil."

Recent additions: tasty-dejafu 1.2.1.0

Added by barrucadu, Sun Jan 20 02:42:46 UTC 2019.

Deja Fu support for the Tasty test framework.

Recent additions: hunit-dejafu 1.2.1.0

Added by barrucadu, Sun Jan 20 02:42:45 UTC 2019.

Deja Fu support for the HUnit test framework.

Recent additions: dejafu 1.12.0.0

Added by barrucadu, Sun Jan 20 02:42:44 UTC 2019.

A library for unit-testing concurrent programs.

Slashdot: Mark Zuckerberg's Mentor 'Shocked and Disappointed' -- But He Has a Plan

Early Facebook investor Roger McNamee published a scathing 3,000-word article adapted from his new book Zucked: Waking Up to the Facebook Catastrophe. Here's just one example of what's left him "shocked and disappointed": Facebook (along with Google and Twitter) has undercut the free press from two directions: it has eroded the economics of journalism and then overwhelmed it with disinformation. On Facebook, information and disinformation look the same; the only difference is that disinformation generates more revenue, so it gets better treatment.... At Facebook's scale -- or Google's -- there is no way to avoid influencing the lives of users and the future of nations. Recent history suggests that the threat to democracy is real. The efforts to date by Facebook, Google and Twitter to protect future elections may be sincere, but there is no reason to think they will do anything more than start a game of whack-a-mole with those who choose to interfere. Only fundamental changes to business models can reduce the risk to democracy. Google and Facebook "are artificially profitable because they do not pay for the damage they cause," McNamee argues, adding that some medical researchers "have raised alarms noting that we have allowed unsupervised psychological experiments on millions of people." But what's unique is he's offering specific suggestions to fix it. "I want to set limits on the markets in which monopoly-class players like Facebook, Google and Amazon can operate. The economy would benefit from breaking them up. A first step would be to prevent acquisitions, as well as cross subsidies and data sharing among products within each platform." "Another important regulatory opportunity is data portability, such that users can move everything of value from one platform to another. This would help enable startups to overcome an otherwise insurmountable barrier to adoption." "Given that social media is practically a public utility, I think it is worth considering more aggressive strategies, including government subsidies." "There need to be versions of Facebook News Feed and all search results that are free of manipulation." "I would like to address privacy with a new model of authentication for website access that permits websites to gather only the minimum amount of data required for each transaction.... it would store private data on the device, not in the cloud. Apple has embraced this model, offering its customers valuable privacy and security advantages over Android." "No one should be able to use a user's data in any way without explicit, prior consent. Third-party audits of algorithms, comparable to what exists now for financial statements, would create the transparency necessary to limit undesirable consequences." "There should be limits on what kind of data can be collected, such that users can limit data collection or choose privacy. This needs to be done immediately, before new products like Alexa and Google Home reach mass adoption."

Read more of this story at Slashdot.

explodingdog: “my favorite snowflake” from explodingdog.com 4 years...



“my favorite snowflake” from explodingdog.com 4 years ago
https://www.instagram.com/explodingdog_sam/p/Bs1jauogbtr/?utm_source=ig_tumblr_share&igshid=1ghc1tonr17ed

Recent additions: network-uri-static 0.1.2.1

Added by snak, Sun Jan 20 00:44:02 UTC 2019.

A small utility to declare type-safe static URIs

Slashdot: Are You Ready For DNS Flag Day?

Long-time Slashdot reader syn3rg quotes the DNS Flag Day page: The current DNS is unnecessarily slow and suffers from inability to deploy new features. To remediate these problems, vendors of DNS software and also big public DNS providers are going to remove certain workarounds on February 1st, 2019. This change affects only sites which operate software which is not following published standards. Are you affected? The site includes a form where site owners can test their domain -- it supplies a helpful technical report about any issues encountered -- as well as suggestions for operators of DNS servers and DNS resolvers, researchers, and DNS software developers. The Internet Systems Consortium blog also has a list of the event's supporters, which include Google, Facebook, Cisco, and Cloudflare, along with some history. "Extension Mechanisms for DNS were specified in 1999, with a minor update in 2013, establishing the 'rules of the road' for responding to queries with EDNS options or flags. Despite this, some implementations continue to violate the rules. "DNS software developers have tried to solve the problems with the interoperability of the DNS protocol and especially its EDNS extension by various workarounds for non-standard behaviors... These workarounds excessively complicate DNS software and are now also negatively impacting the DNS as a whole. The most obvious problems caused by these workarounds are slower responses to DNS queries and the difficulty of deploying new DNS protocol features. Some of these new features (e.g. DNS Cookies) would help reduce DDoS attacks based on DNS protocol abuse.... "Our goal is a reliable and properly functioning DNS that cannot be easily attacked."

Read more of this story at Slashdot.

Slashdot: Venezuela's Government Blocks Access To Wikipedia

Haaretz (with contributions from Reuters and the Associated Press) reports: According to NetBlocks, a digital rights group that tracks restrictions to the internet, as of 12 January, Venezuela largest telecommunications provider CANTV has prevented access to Wikipedia in all languages. The internet observatory told Haaretz the ban was discovered by attempting "to access Wikipedia and other services 60,000 times from 150 different points in the country using multiple providers." Roughly 16 million people have access to the internet in the South American country ravaged by poverty and now facing a political crisis as leader Nicolas Maduro attempts to cling to power following a highly contested re-election last year. Wikipedia receives on average 60 million views from the country every month. According to NetBlocks, the ban was likely imposed after a Wikipedia article listed newly-appointed National Assembly president Juan Guaidà as âoepresident number 51 of the Bolivarian Republic of Venezuela,â ousting Maduro from his presidential status on Wikipedia... Alp Toker, the head of NetBlocks, explained to Haaretz that the block followed a string of controversial edits on the Spanish-language article for Guaido as well as other related articles. Long-time Slashdot reader williamyf identifies himself as "a Venezuelan in Venezuela." He reports that "The method used seems to be to intercept the SSL handshake and not a simple DNS block," adding "the situation is developing." In May of last year the government declared a "state of emergency" that authorized the government to police the internet and filter content, rights activists reported Monday. They added that now Venezuela's new leaders plan to introduce legislation requiring messaging service providers to censor content, and implementing other so-called "content security" measures.

Read more of this story at Slashdot.

Perlsphere: Grant Extension Approved: Jonathan Worthington's Perl 6 Performance and Reliability Engineering grant

Jonathan Worthington recently requested an extension of his Perl 6 Performance and Reliability Engineering grant.

I'm pleased to announce that the Board of Directors approved extension of another $10,000. It'll allow him to dedicate another 167 hours to this work.

I would like to thank the community members who took time to comment on this grant extension request and our sponsors who made funding the grant possible through our Perl 6 Development Fund.

I also appreciate Jonathan Worthington's continued hard work and monthly updates with the grant manager, Matthias Bloch.

Daniel Lemire's blog: Science and Technology links (January 19th, 2019)

  1. Losing even just a bit of weight can be enough to restore fertility in women.
  2. Digital technology does not appear to have a significant negative effect on teenagers, according to an article published by Nature.
  3. According to an article published by Nature, woody vegetation cover over sub-Saharan Africa increased by 8% over the past three decades. This is part of a larger trend: the Earth is getting greener.
  4. The number of children in India has peaked. This suggests that the population of India will peak soon as well.
  5. Scientists are looking for chemical factors that either decrease or increase with age: normalizing these factors could slow or reverse aging. Nature reports on a factor called MANF which decreases with age in mice, human beings and flies. It seems than MANF supplements could have anti-aging benefits.
  6. Netflix is approaching 150 million subscribers. This means that about 2% of the world’s population is made of Netflix subscribers. This almost certainly underestimates the true reach of Netflix.
  7. In an article in the reputed Guardian newspaper from 2004, we can read:

    A secret report, suppressed by US defence chiefs and obtained by The Observer, warns that major European cities will be sunk beneath rising seas as Britain is plunged into a ‘Siberian’ climate by 2020. Nuclear conflict, mega-droughts, famine and widespread rioting will erupt across the world. (…) Already (…) the planet is carrying a higher population than it can sustain. By 2020 ‘catastrophic’ shortages of water and energy supply will become increasingly harder to overcome, plunging the planet into war.

  8. Our brains are made of white and grey matter. Obesity correlates with lower gray matter volume in the brain.
  9. Men and women have (statistically) different cognitive abilities, with women being better at verbal tasks whereas men are better at spatial tasks. This survey concludes that homosexuals have abilities closer to the other sex:

    The meta-analysis revealed that homosexual men performed like heterosexual women in both male-favouring (e.g., spatial cognition) and female-favouring (e.g., verbal fluency) cognitive tests, while homosexual women performed like heterosexual men only in male-favouring tests. The magnitude of the sexual orientation difference varied across cognitive domains (larger for spatial abilities).

Open Culture: Oodles of Classic Doctor Who Episodes Streaming Free Online This Month

A quick fyi: This month, Twitch is presenting a marathon streaming of classic Doctor Who episodes. Continuing through January 25th, they plan to broadcast "11 to 12 hours of new episodes per day (~27 episodes), repeating once so you can catch Doctor Who nearly 24 hours a day, every day..." Stream the episodes right above, or here on Twitch.

Follow Open Culture on Facebook and Twitter and share intelligent media with your friends. Or better yet, sign up for our daily email and get a daily dose of Open Culture in your inbox. 

If you'd like to support Open Culture and our mission, please consider making a donation to our site. It's hard to rely 100% on ads, and your contributions will help us provide the best free cultural and educational materials.

via BoingBoing

Related Content:

30 Hours of Doctor Who Audio Dramas Now Free to Stream Online

The BBC Creates Step-by-Step Instructions for Knitting the Iconic Doctor Who Scarf: A Document from the Early 1980s

Vincent van Gogh Visits a Modern Museum & Gets to See His Artistic Legacy: A Touching Scene from Doctor Who

42 Hours of Ambient Sounds from Blade Runner, Alien, Star Trek and Doctor Who Will Help You Relax & Sleep

The Fascinating Story of How Delia Derbyshire Created the Original Doctor Who Theme

Oodles of Classic Doctor Who Episodes Streaming Free Online This Month is a post from: Open Culture. Follow us on Facebook, Twitter, and Google Plus, or get our Daily Email. And don't miss our big collections of Free Online Courses, Free Online Movies, Free eBooksFree Audio Books, Free Foreign Language Lessons, and MOOCs.

Explosm.net: Comic for 2019.01.19

New Cyanide and Happiness Comic

MattCha's Blog: The Best of the Cheapest: 2018 the Essence of Tea Bamboo Spring



The Essence of Tea usually releases their autumn puerh after Black Friday.  Last year they released a super cheap, and very popular 2017 the Essence of Tea Autumn Nancai which sold out quick and has great reviews.  Although the Essence of Tea is not always known for bargain puerh, it is a good place to look for people who want a pure and non-plantation option.  They released a few other cheaper options a month or so ago.

The 2018 Essence of Tea Yiwu ($0.20/g) is whole year’s production (Spring & Autumn).  This is a very interesting way of keeping the price down while still offering a single garden product.  They also released a 2018 Essence of Tea Spring Wuliang Wild that is a cheaper wild option ($0.20/g), I like their 2017 production as a soft intro for people interested in exploring wild tea.

Anyways…

This 2018 Bamboo Spring is also from the Wuliang producing area.  I like Wuliang puerh for its fresh vibrancy and easiness to drink.  Since Yunnan Sourcing offered its first Wuliang in 2009, I have always had at least some on hand to drink fresh or semi-aged instead of green tea.  I really think Wuliang tea is one of the better regions for drinking now.  I tend to think it peaks at about 10 years, but who am I to know, I have never tried really old Wuliang puerh… Have you?... hahaha.

I recently read an interesting post by Shah8 which broke down and compared all the Wuliang puerh the Essence of Tea has offered over the last few years.  Shah8 thought that this 2018 was a very nice “disposable” drinker with no flaws for its price.

Let’s see how it contends in my search for the best cheapest fresh sheng puerh…

Dry leaves smell of vegetal sweetness, with a distant lingering fruitiness far in the distance.

The first infusion is very light sweet vegetal with a slight grain sweetness base on mild dry wood.  The taste is very light and cheery.  Very green lots of light vegetal notes.  The returning sweetness has a sweet honey dew melon and light honey taste.  The mouthfeel is sticky and the mild fresh fruity taste lingers a minute or so later there is a mild taste like a pink watermelon Jolley Rancher candy that appears.

The second infusion has a grain fruity approach with a grain and dry wood fresh base flavor.  The profile is very light and fresh.  There is a nice cooling almost candy Jolly Rancher and melon returning sweetness and lingering sweet fruity aftertaste.  The mouthfeel is mildly sticky, barely astringent and the tongue a touch sandy.  Qi is mildly relaxing here.

The third infusion has a much more fruity vegetal and even floral initial taste the woody/grain base is very faint almost overwhelmed by the sweeter notes.  The sweet cooling returns nicely and a candy like taste emerges with a grainy base even in the aftertaste.  The aftertaste has some floral in there as well. The mouthfeel is sticky with a sandy tongue sensation.  The profile is overwhelmingly light and green and fresh.  It is vegetal, long minty pungent candy taste with a wood almost grainy base.  It has a very Wuliang profile to it.  I can feel some qi in the eyes and stomach lightly.  The aftertaste lingers in the mouth for a while and even has a subtle rubberiness.

The fourth infusion has an almost bitterness to it initially and has a more vegetal fresh green profile this infusion has more melon and fragrant fruit in a sticky mouthfeel and sandy tongue.  This infusion has a beautiful heavy floral nuance to it.  It’s more Wuliang fruit and has a fuller sweet green taste to it.  The floral perfume lingers long on the breathe.  In a mouthfeel that is slightly oily.

**** then something happened in my first gong fu session of this tea at work.  My work schedule got unexpectedly intense and I had to walk away from the dry leaves for 5 hours.  When I came back the tea leaves had lost all its high noted flavor and the base taste of woods and slight grain was all that remained.  I had a good look at the wet leaf and sure enough it is filled with very tippy, small, buds and leaves.  This tells me that this tea is heavily weighted with odours and more volatile highnotes.  It’s probably a better drink now puerh, very, very light type of puerh- very green but a heavy dosing of these highernotes, for sure.

I one cup the leaves and start from fresh dry leaves the next day…

At the fifth infusion I catch up on my notes… this infusion has a almost gummy candy like vegetal sweetness.  The mouthfeel is nice and sticky and the liquor is slightly viscous.  Long heavy perfumes over an unrecognizable base taste are found here.  The qi is more of a relaxing thing.  Fruity vegetal florals linger long on the breathe.

The sixth infusion delivers an initial taste of heavy perfume florals in a light vegetal base taste.  There is some very mild astringency, almost none.  The sticky mouth coating is nice and throat feel is slightly opening in the mid throat.  This taste extends long in the ftertaste with a low cooling sensation.  Relaxing scalp qi sensation, nice and relaxing vibe.

The seventh infusion has more lighter rolling florals, this tea is packed with these.  The base flavor is so light just fresh vegetal here.  Nice almost spacy qi as it builds like a mist in the mind.  This puerh is pretty gentle in the body so is good enough to drink now- you can feel it stagnate in the stomach a tinny bit only.

The eighth infusion is a bit more vegetal and almost bitter.  The mouthfeel tightens up a bit with the bitter-astringency going on.  There are some heavier florals in the front and longer vegetal florals in the finish.  A bitter green astringency stays in the mouth as well but it is not off putting at all- its just there.  There is also a subtle cinnamon hint which was apparent in my one cup session yesterday.  The qi is a nice relaxing, fluffy qi.

The ninth infusion is more vegetal and wood than floral which mainly trails out in the aftertaste.  The floral, fresh, vegetal aftertaste is still pretty strong in the aftertaste along with cooling notes.  There is a notable dry wood note in the base taste and the liquor is losing its oiliness and is a touch sandy on the tongue.  The astringency can mildly be felt in the upper throat.

The tenth is a woody almost astringent onset with a increasingly grainy tongue feeling.  The cooling aftertaste contains florals but not as brilliant and flanked with some vegetal and woods.

The eleventh is juicier initially like the white of a watermelon then it goes to a woody, slight astringent.  The cooling aftertaste is vegetal, almost dry wood, and slightly sweet.  More vegetal than sweet here.

The twelfth has a nice monotone sweet vegetal light, fresh taste again with the woody and vegetal disappearing here.

The thirteenth infusion is more woody, sticky, with much of the sweet notes just faint in the aftertaste.

I add a good 30+ seconds to the flash in the fourteenth infusion to see what becomes of this tea.  It is pretty much just dry woody vegetal water.  With a sticky lips, sandy tongue, and slight astringent throat with nothing to hold on to.

I put this one for an overnight steeping…

Essence of Tea mentions in the description that 2018 was a better harvest and most teas are tasting better this year.  They have been getting puerh from the same farmer from years now so this claim has some strength.  For sure this is a nice tea, very light, very green, very highnoted, very very clean and fresh in taste and feel.

I wonder how it would compare to 2018 Yunnan Sourcing Wuliang?  I hear this one is also pretty nice and really cheap this year.  I wonder if 2018 has produced a higher quality pick all across the Wuliang area this year?  I should sample it.

So how does it do in the search for the best cheapest fresh sheng?  Pretty good but it isn’t super cheap though and isn’t really something that will age ito something deep, I think.  If you are in to lighter tasting sheng puerh, green tea-esque puerh or drink now puerh this one is going to really satisfy you for this price, I think.

Comparing it to the only other puerh I sampled in a search for the cheapest, this 2018 white2tea Snoozefest- its hard because these are virtually opposite sheng puerh.  2018 Bamboo Spring is single estate, Spring harvest, Wuliang material, and very green and pure, very easy to drink.  2018 Snoozefest is blended (from multiple regions, I suspect), Autumn harvest, maybe not as pure, and needs aging to curb some of the harshness.  These really are totally different beasts and I like them both for their price, it depends if you like more intense profile (white2tea) or gentle profile (the Essence of Tea).

Important to note is that 2018 white2tea Snoozefest sold for $0.08/g and this 2018 the Essence of Tea Bamboo Spring for $0.16/g, double the price.  Overall, the Bamboo Spring wins even when considering the price discrepancy, it’s just a more beautiful puerh simply put…  But since I’m not looking for drink now, I’m looking for something to age in volume and drink later and because it is just so much cheaper (twice as cheap), I would consider the Snoozefest still the best of the cheapest so far for aging.  To be honest, I’m not really excited about either aging into something too amazing and I still don’t think I would re-order either.  But for drinking right now, the 2018 Essence of Tea Bamboo Spring is real nice for the price.  Its one of those thrown one into your cart at check out, kind of things.  That's what I did and my 100g sample is almost gone- the speed test doesn't lie…

Peace

MattCha's Blog: 2018 white2tea Bamboo Shu & Thoughts on Bamboo Puerh


The concept of puerh pressed into a shoot of bamboo sounds gimmicky but, like white2tea’s take on minis, it actually has quite a long history.  Historically, it has been produced for a very long time by the people of Yunnan.  It is sometimes offered to guests in areas where puerh is produced.  I was even gifted some produced in 2001 by tea guests in the mid 2000s.  I busted up one of those bamboo stalks a few years ago to try it out… pretty satisfying and easy drinking sheng puerh.

The production of bamboo puerh is kind of interesting and different than other puerh and is nicely outlined on this blog post here.

It seems like our western puerh vendors have picked up on the novelty of bamboo puerh or maybe it is just starting to get trendy in China?  Either way, everyone seems to be producing and offering it these days.  Yunnan Sourcing has been offering bamboo puerh on their site forever.  Currently they have three kinds available.  I recently saw that the Essence of Tea is producing their own that is not yet released.  And there is of course this 2018 white2tea Bamboo Shu, a New Year release at white2tea which goes for $32.00 for 200g ($0.16/g) of puerh in bamboo.

Here is an interesting older Teachat link on Bamboo shu with Bears, Gingko, and Marshal’N chiming in with their thoughts.  Some of the first bamboo puerh that was offered for sale in the West was a bit low quality- the producers using the novelty of bamboo to hide the actual poor quality of the maocha.  This is probably the reason why it hasn’t gained greater popularity in the West.

Let’s try this free sample out that I received in a recent white2tea re-order…

The dry leaves smell of muted fresh bamboo more than a typical shu puerh odour.  It’s interesting and almost unnatural to smell shu puerh that smells refreshing.  After the rise the wet leaves have a stronger smell like toasted grains and fresh bamboo.  The taste is fresh and roasted tasting as well as deep and milky almost coco shu tasting.  There is a nice coolness on the breath.  The flavor of roasted grains (or bamboo, of course), slight freshness and rich milky mild coco pairs so nicely together and gives this shu puerh lots of depth.  The roast almost gives it a dark roast coffee vibe.  The qi is strong and alerting, I feel it on the skull.  The mouthfeel is decently rich and velvety in the mouth and throat. It feels really nice, mellow, in the body.  This shu puerh is tasty and I love the flavor combination- very satisfying on this -30 C morning.  This is some great shu to drink now while the roasted effects are still active and fresh within the puerh leaves. 

There has been some debate out there whether bamboo puerh ages well.  Some say that the positives of the bamboo roasting are lost in aging and that it can’t age optimally enclosed in bamboo.  I have personally tired some pretty good sheng that was aged from the 90s in bamboo and it tasted good but the roasty bamboo taste was pretty much gone, not fresh and vibrant like this shu. 

Another thing I really like about bamboo puerh is its very compact compression.  For this 2018 white2tea Bamboo Shu, I really like the way the tight compression slowly unravels the fermented shu puerh tastes.  Notice how shu often gives it all up early in the session?  The tight compression allows for a longer balanced gongfu session- I like that about bamboo shu.

Out of all the bamboo puerh I have consumed, none have been this fresh and bamboo tasting.  I think this speaks how quick it has been put to market after the roasting step of production.  Shu is made to drink now, that’s what I would suggest for this one.  This shu is the same price as the 2018 white2tea Smoove Cocoa minis, but the quality and profile of this bamboo shu is probably at least 3 times better than the 2018 Smoove Cocoa which I wasn’t a big fan of.  This one will sell out fast.

Peace
 

ScreenAnarchy: DEMONS: Take A First Look A Daniel Hui's Berlin Selected Dark Satire

Singpore director Daniel Hui wades right in to the thick of the debate around sexual politics, manipulation and abuse of power with his new feature Demons. Following a successful bow at the Busan International Film Festival the film has now been selected as part of the Forum section in Berlin and we're very pleased to share the first trailer for the film here. When Vicki gets a role in Daniel’s new theatre production, she thinks she has gotten the chance of her life. Little does she know this is only the beginning of repeated abuse by Daniel as part of his artistic process. When she turns to the people around her for support, she finds that everyone else seems to think her abuse is beneficial...

[Read the whole post on screenanarchy.com...]

explodingdog: Photo



BOOOOOOOM! – CREATE * INSPIRE * COMMUNITY * ART * DESIGN * MUSIC * FILM * PHOTO * PROJECTS: Booooooom TV Best of the Week: Music Videos, Short Films & Animation

Penny Arcade: News Post: ‘Tis Better

Tycho: Of course, the latest version of the story dutifully presented to is that - let me see if I have this correctly - all games exist in a kind of superposition in a silvery twilight realm, at once real and not real, so the open world game they used to distract us from the previous, higher profile cancellation served its purpose…?  We posit a quantum conundrum in today’s . incomparable strip. For me, Electronic Arts is now the company that pays Respawn to make games and honestly that’s fine.  I guess they also make the sports games that Gabriel plays for two…

ScreenAnarchy: SXSW 2019: Jordan Peele's US, Features, Docs, Episodic TV, and Much More

The 2019 edition of the SXSW Film Festival, which runs from March 8 to March 16th, is right around the corner. SXSW recently announced that opening night film: Jordan Peele's highly anticipated Us will be the opening night film. This week, the festival revealed its lineup of feature films, documentaries, episodic television premieres, and special events. Some of the many programming highlights include world premieres of Harmony Korine's The Beach Bum, Chris Morris' The Day Shall Come, the What We Do In the Shadows television series, and Paul Solet's Tread. The rest of the program, including midnight films and shorts, will be unveiled on February 6th.  A full list of what's been announced so far is provided below. NARRATIVE FEATURE COMPETITION Ten world premieres, ten unique ways to celebrate the art of...

[Read the whole post on screenanarchy.com...]

Colossal: Large-Scale Drawings of the United Kingdom’s 69 Cities by Carl Lavia

Edinburgh, all images provided by Carl Lavia and Lorna Le Bredonchel

Self-taught artist Carl Lavia, who goes by the nickname “Sketch,” has been drawing intricate cities and architecturally-minded illustrations since he was the age of five. Although his early works were imaginative renderings of fictionalized cities, his practice has grown into immensely detailed depictions of large cities from an aerial point of view. Lavia uses ink and archival paper to produce each drawing, which appear like maps from a distance, but have a loose, almost Impressionist style when viewed up close.

In 2016 Lavia paired with photographer Lorna Le Bredonchel to form the project “#69Cities,” which aims to create a portrait of the whole United Kingdom through large-scale drawings of its 69 cities. It takes approximately 2-4 months for Lavia to create each drawing in the expansive project, with Le Bredonchel documenting the entire process from his wall-sized sketches to their time exploring each city by foot.

So far the duo has traveled to Birmingham and Manchester, England as well as Edinburgh, Dundee, Stirling, and Perth, Scotland. The completed Perth drawing will be on display at the Perth Museum and Art Gallery at the end of his month, and be on view through January 2020. When completed, Lavia and Le Bredonchel hope to develop an app that will allow visitors to explore each drawing in an immersive 3D landscape. You can follow along with the pair’s upcoming travels and future drawings on their website, Instagram, and Twitter.

City of Birmingham, all images provided by Carl Lavia and Lorna Le Bredonchel

City of Birmingham, all images provided by Carl Lavia and Lorna Le Bredonchel

Detail of Perth

Detail of Perth

Close up of Manchester

Close up of Manchester

Edinburgh

Edinburgh

 

Completed drawing of Manchester

Completed drawing of Manchester

 

Open Culture: Hear Mary Oliver (RIP) Read Five of Her Poems: “The Summer Day,” “Little Dog’s Rhapsody in the Night,” “Many Miles” and “Night and the River”

Poets get to have strong opinions about what poetry should be and do, especially poets as well-loved as Mary Oliver, who passed away yesterday at the age of 83. “Poetry, to be understood, must be clear,” she told NPR in an interview, “It mustn’t be fancy…. I always feel that whatever isn’t necessary should not be in the poem.” Oliver’s Zen approach to her art cut right to the heart of things and honored natural, unpretentious expression. “I don’t know exactly what a prayer is,” she writes in “The Summer Day,” “I do know how to pay attention.”

For Oliver that meant giving careful heed to the natural world, shearing away abstraction and obfuscation. She grew up in Ohio, and during a painful childhood walked through the woods for solace, where she began writing her first poems.

She became an “indefatigable guide to the natural world,” as Maxine Kumin wrote, and at the same time, to the spiritual. She has been compared to Emerson and wrote “about old-fashioned subjects—nature, beauty, and worst of all, God,” Ruth Franklin remarks with irony in a New Yorker review of the poet’s last, 2017 book, Devotions. But, like Emerson, Oliver was not a writer of any orthodoxy or creed.

Oliver’s approach to the spiritual is always rooted firmly in the natural. Spirit, she writes, “needs the body’s world… to be more than pure light / that burns / where no one is.” She was beloved by millions, by teachers, writers, and celebrities. (She was once interviewed by Maria Shriver in an issue of magazine; Gwyneth Paltrow is a big fan). Oliver was long the country’s best-selling poet, as Dwight Garner blithely writes at The New York Times. But “she has not been taken seriously by most poetry critics,” Franklin points out. This despite the fact that she won a Pulitzer Prize in 1984 for her fifth book, American Primitiveand a National Book Award in 1992 for New and Selected Poems.

The word “earnest” comes up often as faint praise in reviews of Oliver’s poetry (Garner tidily sums up her work as “earnest poems about nature”). The implication is that her poems are slight, simple, unrefined. This perhaps inevitably happens to accessible poets who become famous in life, but it is also a serious misreading. Oliver's work is full of paradoxes, ambiguities, and the hard wisdom of a mature moral vision. She is “among the few American poets,” critic Alicia Ostriker writes, “who can describe and transmit ecstasy, while retaining a practical awareness of the world as one of predators and prey.” In her work, she faces suffering with “cold, sharp eyes,” confronting “steadily,” Ostriker goes on, “what she cannot change.”

Her poems have included “historical and personal suffering,” but more often she engages the life and death going on all around us, which we rarely take notice of at all. She peers into the darkness of hermit crab shells, she feeds a grasshopper sugar from the palm of her hand, watching the creature’s “jaws back and forth instead of up and down.” Oliver often wrote about the constant reminders of death in life in poems like “Death at a Great Distance” and “When Death Comes.” She wrote just as often about how astonishing it is to be alive when we make deep connections with the natural world.

“When it’s over,” Oliver writes in "When Death Comes," ” I want to say all my life / I was a bride married to amazement.” The cost of not paying attention, she suggests, is to be a tourist in one’s own life and to never be at home. “I don’t want to end up simply having visited this world." In the videos here, see and hear Oliver read “The Summer Day,” “Wild Geese,” “Little Dog’s Rhapsody in the Night,” “Night and the River” (above) and "Many Miles."

Oliver was an artist, says Franklin, “interested in following her own path, both spiritually and poetically,” and in her work she will continue to inspire her readers to do the same. These readings will be added to our collection, 900 Free Audio Books: Download Great Books for Free.

Related Content:

Hear Meryl Streep Read Sylvia Plath’s “Morning Song,” a Poem Written After the Birth of Her Daughter

An 8-Hour Marathon Reading of 500 Emily Dickinson Poems

Hear Dylan Thomas Recite His Classic Poem, “Do Not Go Gentle Into That Good Night”

Josh Jones is a writer and musician based in Durham, NC. Follow him at @jdmagness

Hear Mary Oliver (RIP) Read Five of Her Poems: “The Summer Day,” “Little Dog’s Rhapsody in the Night,” “Many Miles” and “Night and the River” is a post from: Open Culture. Follow us on Facebook, Twitter, and Google Plus, or get our Daily Email. And don't miss our big collections of Free Online Courses, Free Online Movies, Free eBooksFree Audio Books, Free Foreign Language Lessons, and MOOCs.

ScreenAnarchy: Friday One Sheet: HIGH LIFE Essays Biology in Space

Thus far, 2019 has been the year of Green. While two weeks hardly make a trend, it cannot be understated how underused green is in movie key art. Here is the poster for Claire Denis's 'sex-ain't-sexy' space drama, High Life, in which Robert Pattinson is raising a child on an abandoned space-ship cum laboratory (pun intended). We witness the results of an experiment using convicts to study reproduction and sterility in deep space exploration has gone awry.  (This is somewhat serendipitous in light of the news that the Chinese Goverment has both put the first plant on the moon, and it died within 24 hours.) So biology and event horizons are on the brain with this one. Highlife is a juxtaposition of opposites, expressed in...

[Read the whole post on screenanarchy.com...]

Open Culture: Vintage Geological Maps Get Turned Into 3D Topographical Wonders

What good is an old-fashioned map in the age of apps?

One need not be a mountaineer, geoscientist, or civil engineer to get the topographical lay of the land with a speed and accuracy that would have blown Lewis and Clark’s minds’ right through the top of the lynx and otter toppers they took to wearing after their standard issue army lids wore out.

There’s still something to be said for the old ways, though.

Graphic designer Scott Reinhard has all the latest technological advances at his disposal, but it took combining them with hundred-year-old maps for him to get a truly 3-D appreciation for locations he has visited around the United States, as well as his childhood home.

A son of Indiana, Reinhard told Colossal’s Kate Sierzputowski that he found some Grand Teton-type excitement in the notoriously flat Hoosier State once he started marrying official national geospatial data to vintage map designs:

 When I began rendering the elevation data for the state, the story of the land emerged. The glaciers that receded across the northern half of the state after the last ice age scraped and gouged and shaped the land in a way that is spectacularly clear…I felt empowered by the ability to collect and process the vast amounts of information freely available, and create beautiful images.

(The government shut-down has not damaged the accuracy of Reinhard’s maps, but the U.S. Geological Survey’s website does warn the public that the effects of any earthquakes or other force majeure occurring during this black-out period will not immediately be reflected in their topos.)

(Nor are they able to respond to any inquiries, which puts a damper on holiday weekend plans for making salt dough maps, another Hoosier state fave, at least in 1974...)

As writer Jason Kottke notes, the shadows the mountains cast on the margins of Reinhard’s maps are a particularly effective optical trick.

You can see more of Reinhard’s digitally enhanced maps from the late 19th and early 20th-century, and order prints in his online shop.

via Kottke/Colossal

Related Content:

View and Download Nearly 60,000 Maps from the U.S. Geological Survey (USGS)

Download 67,000 Historic Maps (in High Resolution) from the Wonderful David Rumsey Map Collection

The History of Cartography, “the Most Ambitious Overview of Map Making Ever Undertaken,” Is Free Online

Ayun Halliday is an author, illustrator, theater maker and Chief Primatologist of the East Village Inky zine.  See her onstage in New York City in February as host of  Theater of the Apes book-based variety show, Necromancers of the Public Domain. Follow her @AyunHalliday.

Vintage Geological Maps Get Turned Into 3D Topographical Wonders is a post from: Open Culture. Follow us on Facebook, Twitter, and Google Plus, or get our Daily Email. And don't miss our big collections of Free Online Courses, Free Online Movies, Free eBooksFree Audio Books, Free Foreign Language Lessons, and MOOCs.

Colossal: Graphically Designed Ceramic Vessels Form Zoetrope Animations When Spun on a Pottery Wheel

After years of work as a ceramic artist, Kenny Sing of Turn Studio has created a series of shallow vessels which double as zoetrope animations when spun. His project, Trepō, transfers digital patterns onto the one-of-a-kind curves of his ceramic platters. The patterns are then either precisely cut from or glazed onto their surface. These elements act as static designs until they are activated by a pottery wheel. As the wheel turns, the patterns come to life: cubes, triangles, and rectangles appear to tumble into the center of the vessel. You can view the process for creating one of the ceramic vessels in the video below, and view more works on Turn Studio’s website, Instagram, and Vimeo. (via Colossal Submissions)

BOOOOOOOM! – CREATE * INSPIRE * COMMUNITY * ART * DESIGN * MUSIC * FILM * PHOTO * PROJECTS: Illustrator Spotlight: Tatsuro Kiuchi

Tatsuro Kiuchi

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi

 

 

Tatsuro Kiuchi’s Website

Tatsuro Kiuchi on Instagram

Michael Geist: Sunlight on the Submissions: Why the Broadcasting and Telecommunications Legislative Review Panel Should Reverse Its Secretive Approach

The Broadcasting and Telecommunications Legislative Review panel’s surprising decision to keep the 2,200 public submissions secret for months has had immediate repercussions. Some organizations are refusing to disclose their submissions until the panel does and others have noted the missed opportunity for public discussion of a vitally important issue. To date, about 30 submissions have been posted, a tiny percentage of the total. The decision has had an impact on university courses and predictably created an information asymmetry with some companies cherry-picking who gets to see their submission.

The approach runs counter to the government’s support for open, transparent policy making processes and the standard used within the sector by the CRTC. When one report indicated that the secrecy was at the request of ISED Minister Navdeep Bains, chair Janet Yale issued a public statement:

Any suggestion of ministerial direction concerning the release of written submissions made to the Panel is flatly incorrect. All submissions will be published along with the Panel’s ‘What We’ve Heard Report’ to be released no later than June 30th. That decision was taken by the panel alone as is to be expected of an independent, arms-length panel. Interested parties are free to make their submissions public.

This represents a change in language from the panel’s own press release on the consultation, which stated “these written submissions will be publicly available after the deadline for submissions on November 30, 2018.” The release did not say submissions would only be available once the panel’s interim report was released and few reasonably thought that “after the deadline” was a half-a-year or more after.

I followed up with a request for an explanation of why the panel has chosen secrecy over public disclosure. The response in full:

The Panel wants to take time to read, review, and digest the submissions, and will make all submissions public when it releases the What We’ve Heard Report. Interested parties are welcome to make their submissions public and many have already done so.

Here is their initial statement clarifying:

January 11th marked the deadline for written submissions to the Panel – this is the extended deadline we set after accommodating requests for more time to prepare submissions.

The Panel members wish to extend our sincere appreciation to the 2,000 interested parties who took the time and dedicated the effort to prepare submissions.

In addition to the submissions, the Panel met with over 100 interested parties.

We will now take a period of time to review and evaluate these submissions, along with our other outreach activities. These inputs will form the backbone of our What We’ve Heard Report which we plan to publish no later than June 30th. At that time, all written submissions will also be posted on the Panel’s website.

This response doesn’t really answer the question since the panel can obviously “read, review and digest the submissions” while the submissions are public and being assessed by those interested in communications policy.

Yet the decision is more troubling for what it says about the panel (which, it must be said, brings decades of experience and has committed an enormous amount of time to this challenging work). Diverting from established transparency norms should not happen without a strong justification. In this case, at best the panel seems to suggest that public discussion, debate, or even rebuttals of submissions might distract from its efforts to engage in a fair assessment process.

This hardly represents a strong justification. First, such a response would be viewed as entirely unacceptable by CRTC commissioners, politicians, regulators, or government policy makers, who all frequently engage in policy review with a public record readily available (less redacted confidential information). If panelists do not want to consider public debate on submissions, they can mute the discussion on social media or ignore public postings and media coverage. Establishing a panel publication-style ban on submissions for months goes far beyond what is necessary for their assessment process.

Second, the panel’s report in June is only a “what we’ve heard” report, not the report’s actual recommendations. There will be no insulating the panel from public commentary about the submissions and perspectives in the period from the initial report in June 2019 to the final report in 2020. Indeed, if the panel has concerns about coping with public debate about submissions during the development of a factual report summarizing its meetings and public submissions, how will it handle the more critical phase that leads to its recommendations?

The submissions that are currently available provide a pretty wide range of perspectives, but without Bell, Rogers, Netflix, Facebook, the CMPA, dozens of other organizations, and hundreds of individuals, it is incomplete. The panel should reconsider its ill-advised approach and move to quickly release all submissions without further delay.

The post Sunlight on the Submissions: Why the Broadcasting and Telecommunications Legislative Review Panel Should Reverse Its Secretive Approach appeared first on Michael Geist.

OCaml Planet: Deprecating file releases on the forge

As part of the deprecation plan for forge.ocamlcore.org, the ability to upload files to the forge will soon be deprecated. Projects will not have the ability to create new releases. I will keep the current uploaded files and will continue to serve them. Here is what will change, once the deprecation will be done: - files will be moved to a new static host: https://download.ocamlcore.org - all links in the "Files" section will point to an URL https://download.ocamlcore.org - statistics about number of downloads for files will be removed everywhere on the site - redirection from old URL to new ones will be maintained as a best effort. - URLs in OPAM will change to point to https://download.ocamlcore.org

Penny Arcade: Comic: ‘Tis Better

New Comic: ‘Tis Better

ScreenAnarchy: Trailer: Jonas Åkerlund's LORDS OF CHAOS Jumps Into The Fire That Forged The Blackest Metal

A film nearly a decade in the making, Jonas Åkerlund's Lords of Chaos is finally set to hit theaters in North America on February 8th from indie distributor Gunpowder & Sky. To promote their new film, the first domestic trailer for the film dropped today and it does an amazingly good job of conveying the really weird tonal dissonance of the film which enjoyed a healthy festival run in 2018. Lords of Chaos is based on the 1997 non-fiction book by Michael Moynihan, which detailed the rise and fall of Norwegian Black Metal and in particular the bloody chaos that followed the band Mayhem and some of its founding members. The film stars Rory Culkin as Euronymous, the founder of Mayhem, along with a strong supporting...

[Read the whole post on screenanarchy.com...]

BOOOOOOOM! – CREATE * INSPIRE * COMMUNITY * ART * DESIGN * MUSIC * FILM * PHOTO * PROJECTS: Artist Spotlight: Samuel Rodriguez

Samuel Rodriguez (previously featured here).

Samuel Rodriguez

 

 

Samuel Rodriguez

 

 

Samuel Rodriguez

 

 

Samuel Rodriguez

 

 

Samuel Rodriguez

 

 

Samuel Rodriguez

 

 

Samuel Rodriguez

 

 

Samuel Rodriguez

 

 

Samuel Rodriguez

 

 

Samuel Rodriguez

 

 

Samuel Rodriguez’s Website

Samuel Rodriguez on Instagram

Open Culture: Hear the Sounds of the Actual Instruments for Which Mozart, Beethoven, Haydn, and Handel Originally Composed Their Music

When we go to a concert of orchestral music today, we hear most every piece played on the same range of instruments — instruments we know and love, to be sure, but instruments designed and operated within quite strict parameters. The pleasing quality of the sounds they produce may make us believe that we're hearing everything just as the composer originally intended, but we usually aren't. To hear what the likes of Mozart, Beethoven, Handel, and Haydn would have had in their head as they composed back in their day, you'd have to have an orchestra go so far as to play it not with modern instruments, but the same ones orchestras used back in those composers' lifetimes.

Enter London's Orchestra of the Age of the Enlightenment, which takes its name from the era of the late 18th century from which it draws most of its repertoire — and from which it draws most of its instruments, a vital part of its mission to achieve period-accurate sound. You can read more about the OAE's instruments on its web site, or better yet, head over to its Youtube channel to hear those instruments demonstrated and their historical backgrounds explained. Here we have four of the OAE's videos: on the clarinet they use for Mozart's Clarinet Concerto, on the contrabassoon they use for Beethoven's Fifth Symphony and Haydn's Creation, the organ they use for Handel's Organ Concerto, and an oboe like the one Haydn would have known.

"We love the music we play," says OAE double bassist Cecelia Bruggemeyer, "and we love asking questions about the music we play." So when you use an instrument like the 300-year-old bass she shows off in another video, "you suddenly find it doesn't necessarily do the things a modern instrument will do, and that sets up a whole train of questions." These include, "What would Bach have heard? How might the players in his day have played? What does that mean for us, playing today? What does that mean for live music now, with this historic information? We're not trying to re-create the past. We're trying to make something that's exciting now but using what was from the past" — not a bad metaphor, come to think of it, for the entire enterprise of classical-music performance in the 21st century.

Related Content:

What Ancient Greek Music Sounded Like: Hear a Reconstruction That is ‘100% Accurate’

Watch a Musician Improvise on a 500-Year-Old Music Instrument, The Carillon

Visit an Online Collection of 61,761 Musical Instruments from Across the World

Musician Plays the Last Stradivarius Guitar in the World, the “Sabionari” Made in 1679

Hear a 9,000 Year Old Flute—the World’s Oldest Playable Instrument—Get Played Again

The Musical Instruments in Hieronymus Bosch’s The Garden of Earthly Delights Get Brought to Life, and It Turns Out That They Sound “Painful” and “Horrible”

Based in Seoul, Colin Marshall writes and broadcasts on cities, language, and culture. His projects include the book The Stateless City: a Walk through 21st-Century Los Angeles and the video series The City in Cinema. Follow him on Twitter at @colinmarshall or on Facebook.

Hear the Sounds of the Actual Instruments for Which Mozart, Beethoven, Haydn, and Handel Originally Composed Their Music is a post from: Open Culture. Follow us on Facebook, Twitter, and Google Plus, or get our Daily Email. And don't miss our big collections of Free Online Courses, Free Online Movies, Free eBooksFree Audio Books, Free Foreign Language Lessons, and MOOCs.

ScreenAnarchy: Review: IO, A Philosophy of Existence and Connection

It is not easy to know when to have the courage to change something, and when to have the wisdom to accept what can’t be changed. Or even if something can be changed, when it might be too late to make a difference. Do we accept that even our best will not be enough, and move on, or do we keep trying even if such efforts are futile and might end in our death? And what would it matter if it leaves us bereft and alone, if the point of human life is what we find in each other? Sam (Margaret Qualley) is a scientist, and one of the last humans left on a dying Earth. Everyone else has left for IO, one of Jupiter’s...

[Read the whole post on screenanarchy.com...]

Colossal: Dried Grass and Branches are Woven and Stitched into Delicate Sculptural Drawings by Kazuhito Takadoi

Artist Kazuhito Takadoi uses natural materials combined with traditional Japanese art supplies like sumi ink and washi paper to make delicate sculptural works that tread between two and three dimensions. Takadoi cultivates and then gathers grass, leaves, and twigs from his garden to form the meticulous structures that comprise his dimensional drawings. Though these organic findings are secured in place through weaving and stitching, they continue to evolve as they dry and mature, changing in flexibility and color.

The artist, who is originally from Nagoya, Japan, trained in agriculture and horticulture in Japan, the US, and the UK before studying art and garden design in the UK. Takadoi is now based in Britain and is represented by jaggedart gallery in London. His work will be on view at Saatchi Gallery as part of Collect, an international modern craft and design fair, on view in London from February 23 until March 3, 2019.

Planet Haskell: Michael Snoyman: Kids Coding, Part 4

Previous lessons have all been focused on teaching our ten and eight year olds some coding, since our six year old (Yakov) is still working on reading and writing in English. However, Yakov’s been home sick all week, and he asked me today to teach him some programming. So I came up with a simplified version focused solely on the GHCi prompt.

I started off with:

> x = 32
> x + x

And then I asked him what he thought the answer would be. He quickly came back with 64. Then I typed in:

> y = 5
> x + y

And he got 37 pretty quickly. We played around with a few more bits of simple arithmetic (he hasn’t really mastered multiplication yet). I introduced defining variables in terms of other variables:

> x = 2 * 3
> y = x * 2
> y + 3

This took him a bit longer, but entirely due to the multiplication! This showed me a few things:

  1. Reassigning variables was not confusing for him in the least
  2. Variable replacement was similarly trivial

Finally, I decided to push things just a bit further and introduce functions:

> f x = x + 3
> f 7

This confused him at first, but once I explained that this was applying the function f to the number 7, he got it, and said “oh, it’s the +3 function.” (Remember from last time that he’s been playing the function game for a while.) Next I hit:

> x = f 0
> f x

This was easy: it’s 6! Finally I gave him two more challenging ones:

> f (f 0)
> f (f (f 10))

I fully expected confusion about parentheses. I was shocked: he wasn’t bothered by them at all. He immediately got both answers, and was very proud of himself.

Total time: less than 10 minutes, probably closer to 5. Which is good, because he’s got a short attention span and wanted to play some Nintendo with me too. Overall, I was very happy with how many concepts he was able to absorb.

(Thanks to my ~/.ghc/ghci_history file for reminding me what we covered today.)

things magazine: Dog days

Failed Architecture asks What this MVRDV Rendering Says About Architecture and the Media, a musing on the digital delusions of modern architectural imagery. The architects reply: In Defense of Renders and Trees On Top of Skyscrapers. Both sides have valid … Continue reading

Perlsphere: Ranking of Perl sites 2019 January

Once in a while I check the ranking of various Perl-related web sites using Alexa. Today, January 18, 2019, I checked cpan.org and I was surprised to see that it is still in a rather good place: 165,690.

For the full article visit Ranking of Perl sites 2019 January

Explosm.net: Comic for 2019.01.18

New Cyanide and Happiness Comic

The Shape of Code: Choosing between two reasonably fitting probability distributions

I sometimes go fishing for a probability distribution to fit some software engineering data I have. Why would I want to spend time fishing for a probability distribution?

Data comes from events that are driven by one or more processes. Researchers have studied the underlying patterns present in many processes and in some cases have been able to calculate which probability distribution matches the pattern of data that it generates. This approach starts with the characteristics of the processes and derives a probability distribution. Often I don’t really know anything about the characteristics of the processes that generated the data I am looking at (but I can often make what I like to think are intelligent guesses). If I can match the data with a probability distribution, I can use what is known about processes that generate this distribution to get some ideas about the kinds of processes that could have generated my data.

Around nine-months ago, I learned about the Conway–Maxwell–Poisson distribution (or COM-Poisson). This looked as-if it might find some use in fitting software engineering data, and I added it to my list of distributions to keep in mind. I saw that the R package COMPoissonReg supports the fitting of COM-Poisson distributions.

This week I came across one of the papers, about COM-Poisson, that I was reading nine-months ago, and decided to give it a go with some count-data I had.

The Poisson distribution involves count-data, i.e., non-negative integers. Lots of count-data samples are well described by a Poisson distribution, and it is one of the basic distributions supported by statistical packages. Processes described by a Poisson distribution are memory-less, in that the probability of an event occurring are independent of when previous events occurred. When there is a connection between events, the Poisson distribution is not such a good fit (depending on the strength of the connection between events).

While a process that generates count-data may not meet the requirements needed to be exactly described by a Poisson distribution, the behavior may be close enough to give good-enough results. R supports a quasipoisson distribution to help handle the ‘near-misses’.

Sometimes count-data has a distribution that looks nothing like a Poisson. The Negative-binomial distribution is the obvious next choice to try (this can be viewed as a combination of different Poisson distributions; another combination is the Poisson inverse gaussian distribution).

The plot below (from a paper analyzing usage of record data structures in Racket; Tobias Pape kindly sent me the data) shows the number of Racket structure types that contain a given number of fields (red pluses), along with lines showing fitted Negative binomial and COM-Poisson distributions (code+data):

Number of Racket structure types containing a given number of fields.

I’m interested in understanding the processes that are generating the data, and having two distributions do such a reasonable job of fitting the data has given me more possible distinct explanations for what is going on than I wanted (if I were interested in prediction, then either distribution looks like it would do a good-enough job).

What are the characteristics of the processes that generate data having each of the distributions?

  • A Negative binomial can be viewed as a combination of Poisson distributions (the combination having a Gamma distribution). We could create a story around multiple processes being responsible for the pattern seen, with each of these processes having the impact of a Poisson distribution. Sounds plausible.
  • A COM-Poisson distribution can be viewed as a Poisson distribution which is length dependent. We could create a story around the probability of a field being added to a structure type being dependent on the number of existing fields it contains. Sounds plausible (it’s a slightly different idea from preferential attachment).

When fitting a distribution to data, I usually go with the ‘brand-name’ distributions (i.e., the one with most name recognition, provided it matches well enough; brand names are an easier sell then the less well known names).

The Negative binomial distribution is the brand-name here. I had not heard of the COM-Poisson distribution until nine-months ago.

Perhaps the authors of the Racket analysis paper will come up with a theory that prefers one of these distributions, or even suggests another one.

Readers of my evidence-based software engineering book need to be aware of my brand-name preference in some of the data fitting that occurs.

OCaml Planet: Coq 8.9.0 is out

The 8.9.0 release of Coq is available.

It features many quality-of-life improvements, including goal differences as well as numeral and custom notations. For details, see the CHANGES file.

Feedback and bug reports are extremely welcome.

Planet Haskell: Neil Mitchell: Ignoring HLint

Summary: HLint now has more ways to ignore hints you don't like.

HLint makes suggestions about how to improve your Haskell code. But not everyone likes all the suggestions all the time, so HLint comes with ways of ignoring those annoying hints, and HLint 2.1.11 provides even more mechanisms. Without further ado, let's take a quick tour - full details are in the HLint README.

Method 1: the --default flag

To ignore all hints your code currently generates run hlint as normal, but passing the --default flag, which will generate a config file with all hints that fire set to ignore. Typically, when approaching a new code base to run HLint on, I start by doing:

hlint . --default > .hlint.yaml

After that, it's easy to remove ignored hints from .hlint.yaml one by one and fix the code.

Method 2: Add -ignore directives

In the .hlint.yaml file you can write:

- ignore: {name: Eta reduce}

This directive ignores the named hint, and is what --default generates. There are also more refined ways of ignoring a hint in certain modules, or ignoring all hints in certain modules (see the README).

Method 3: Add a {- comment -}

Method 3 actually has 3 sub-methods, you can write any of:

  • {-# ANN module "HLint: ignore Eta reduce" #-}
  • {-# HLINT ignore "Eta reduce" #-}
  • {- HLINT ignore "Eta reduce" -}

For ANN pragmas it is important to put them after any import statements. If you have the OverloadedStrings extension enabled you will need to give an explicit type to the annotation, e.g. {-# ANN module ("HLint: ignore Eta reduce" :: String) #-}. The ANN pragmas can also increase compile times or cause more recompilation than otherwise required, since they are evaluated by TemplateHaskell.

For {-# HLINT #-} pragmas GHC may give a warning about an unrecognised pragma, which can be supressed with -Wno-unrecognised-pragmas.

For {- HLINT -} comments they are likely to be treated as comments in syntax highlighting, which can lead to them being overlooked.

My current preference is {- HLINT -}, but I think GHC should just special case {-# HLINT #-} and then in a few GHC releases we could use that. Unfortunately, other people disagree with me, so {- HLINT -} is the best we have.

Method 4: Using the C Pre Processor

hlint defines the __HLINT__ preprocessor definition (with value 1), so problematic definitions (including those that don't parse) can be hidden with:

#ifndef __HLINT__
foo = ( -- HLint would fail to parse this
#endif

Colossal: The Moon’s Magical Mythology Captured in an Illustrated Book by David Álvarez

In Noche Antigua (Ancient Night) an opossum and a rabbit work together—and against each other—to create and maintain the sun and the moon. The book, written in Spanish and illustrated by Mexico-based artist David Álvarez (previously) is based on elements from ancient myths in several Central American cultures. Álvarez captures a sense of quiet magic with the simplified forms and hushed tones of his illustrations, which seem to glow from the illumination of the moon. You can see more of the artist’s work on Instagram and his Etsy shop, and find a hardcover copy of Noche Antigua on Amazon.

Disquiet: Disquiet Junto Project 0368: Engage Duo

Each Thursday in the Disquiet Junto group, a new compositional challenge is set before the group’s members, who then have just over four days to upload a track in response to the assignment. Membership in the Junto is open: just join and participate. (A SoundCloud account is helpful but not required.) There’s no pressure to do every project. It’s weekly so that you know it’s there, every Thursday through Monday, when you have the time.

Deadline: This project’s deadline is Monday, January 21, 2019, at 11:59pm (that is, just before midnight) wherever you are on. It was posted around noon, California time, on Thursday, January 17, 2019.

These are the instructions that went out to the group’s email list (at tinyletter.com/disquiet-junto):

Disquiet Junto Project 0368: Engage Duo
The Assignment: Record the second third of a trio, adding to a pre-existing track.

Step 1: This week’s Disquiet Junto project is the second in a sequence that explores and encourages asynchronous collaboration. This week you will be adding music to a pre-existing track, which you will source from the previous week’s Junto project (disquiet.com/0367). Note that you aren’t creating a duet — you’re creating the second third of what will eventually be a trio. Keep this in mind.

Step 2: The plan is for you to record a short and original piece of music, on any instrumentation of your choice, as a complement to the pre-existing track. First, however, you must select the piece of music to which you will be adding your own music. There are more than 40 tracks to choose from, almost all of them in this playlist:

https://soundcloud.com/disquiet/sets/disquiet-junto-project-0367

And at least one more is available as part of a video in this discussion thread from last week:

https://llllllll.co/t/disquiet-junto-project-0367-trio-initiate/

To select a track, you can listen through all that and choose one, or you can use a random number generator. (Note: it’s perfectly fine if more than one person uses the same original track as the basis for their piece.)

Step 3: Record a short piece of music, roughly the length of the piece of music you selected in Step 2. Your track should complement the piece from Step 2, and leave room for an eventual third piece of music. When composing and recording your part, do not alter the original piece of music at all, except to pan the original fully to the left. In your finished audio track, your part should be panned fully to the right. To be clear: the track you upload won’t be your piece of music alone; it will be a combination of the track from Step 2 and yours.

Step 4: Also be sure, when done, to make the finished track downloadable, because it will be used by someone else in a subsequent Junto project.

Seven More Important Steps When Your Track Is Done:

Step 1: Include “disquiet0368” (no spaces or quotation marks) in the name of your track.

Step 2: If your audio-hosting platform allows for tags, be sure to also include the project tag “disquiet0368” (no spaces or quotation marks). If you’re posting on SoundCloud in particular, this is essential to subsequent location of tracks for the creation a project playlist.

Step 3: Upload your track. It is helpful but not essential that you use SoundCloud to host your track.

Step 4: Post your track in the following discussion thread at llllllll.co:

https://llllllll.co/t/disquiet-junto-project-0368-engage-duo/

Step 5: Annotate your track with a brief explanation of your approach and process.

Step 6: If posting on social media, please consider using the hashtag #disquietjunto so fellow participants are more likely to locate your communication.

Step 7: Then listen to and comment on tracks uploaded by your fellow Disquiet Junto participants.

Additional Details:

Deadline: This project’s deadline is Monday, January 21, 2019, at 11:59pm (that is, just before midnight) wherever you are on. It was posted around noon, California time, on Thursday, January 17, 2019.

Length: Your finished track will be roughly the length of the track you chose to add to.

Title/Tag: When posting your track, please include “disquiet0368” in the title of the track, and where applicable (on SoundCloud, for example) as a tag.

Upload: When participating in this project, post one finished track with the project tag, and be sure to include a description of your process in planning, composing, and recording it. This description is an essential element of the communicative process inherent in the Disquiet Junto. Photos, video, and lists of equipment are always appreciated.

Download: Please for this project be sure to set your track as downloadable and allowing for attributed remixing (i.e., a Creative Commons license permitting non-commercial sharing with attribution, allowing for derivatives).

For context, when posting the track online, please be sure to include this following information:

More on this 368th weekly Disquiet Junto project (Engage Duo / The Assignment: Record the second third of a trio, adding to a pre-existing track.) at:

https://disquiet.com/0368/

More on the Disquiet Junto at:

https://disquiet.com/junto/

Subscribe to project announcements here:

http://tinyletter.com/disquiet-junto/

Project discussion takes place on llllllll.co:

https://llllllll.co/t/disquiet-junto-project-0368-engage-duo/

There’s also on a Junto Slack. Send your email address to twitter.com/disquiet for Slack inclusion.

Image associated with this project is adapted (edited, color altered, text added) from a photo by Anna J and is used via Flickr thanks to a Creative Commons license:

https://flic.kr/p/gcojn

https://creativecommons.org/licenses/by-nc-sa/2.0/

Open Culture: Behold Moebius’ Many Psychedelic Illustrations of Jimi Hendrix


The 1995 release of posthumous Jimi Hendrix compilation Voodoo Soup has divided fans and critics for over two decades now. But whatever its merits, its cover art should hold an honored place in every Hendrix fan’s collection. Drawn by the legendary cult comic artist Moebius from a photograph of Hendrix eating soup in France , it captures the sound Hendrix was moving toward at the end of his life—his head exploding in flames, or mushroom clouds, or pink psychedelic bronchial tubes. The image comes from a larger gatefold, excerpted below, which Moebius drew for the French double LP Are You Experienced/Axis: Bold as Love in 1975.

Journalist Jean-Nöel Coghe was supposedly very upset that he did not even receive mention for taking the original photo, but in the nineties he and Moebius came together again for a project that would do them both credit, a book called Emotions électriques that Coghe wrote of his experiences traveling through France as Hendrix’s guide during the Experience’s first tour of the country in 1967.

Moebius provided the book's illustrations, many of which you can see below, “each of them,” as the publisher's description has it, “imagining Hendrix in a classic Moebius landscape of dreams.”

 

Obviously a huge Hendrix fan, Moebius is in many ways as responsible for the psychedelic space race of the 1970s as the guitarist himself. His work in the French comic magazine Métal hurlantHeavy Metal in the American version—epitomized the sci-fi and fantasy elements that came to dominate heavy rock. His work with Alejandro Jodorowsky on the Chilean visionary filmmaker’s aborted Dune is the stuff of legend.

Moebius had illustrated album covers since the early seventies, mostly those of European artists. But his creations as a magazine and comics illustrator (and film scenarist) have the most enduring appeal for much the same reason as Hendrix’s music. They are both unparalleled masters and natural storytellers whose imagined worlds are so richly detailed and consistently surprising they have birthed entire genres. The two may have crossed paths too late to actually work together, but I like to think Moebius carried on the spirit of Hendrix in a visual form.

It may not be common knowledge that Hendrix hated his album covers, leaving detailed notes about them for his record company, who ignored them. His own choices, one must admit, including a Linda McCartney photo for the cover of Electric Ladyland that makes the band look like they’re on the set of a proto-Sesame Street, do not exactly sell the records’ treasures. But Jimi might have loved Moebius’ interpretations of his headspace, a visual continuation of a prominent strand of Hendrix's imagination. See all of Moebius' Hendrix illustrations here.

 

Related Content:

Métal hurlant: The Hugely Influential French Comic Magazine That Put Moebius on the Map & Changed Sci-Fi Forever

Iconic Footage of Jimi Hendrix Playing “Hey Joe” Rendered in the Style of Moebius, with the Help of Neural Network Technology

Jimi Hendrix’s Final Interview Animated (1970)

Josh Jones is a writer and musician based in Durham, NC. Follow him at @jdmagness

Behold Moebius’ Many Psychedelic Illustrations of Jimi Hendrix is a post from: Open Culture. Follow us on Facebook, Twitter, and Google Plus, or get our Daily Email. And don't miss our big collections of Free Online Courses, Free Online Movies, Free eBooksFree Audio Books, Free Foreign Language Lessons, and MOOCs.

Planet Haskell: FP Complete: When Rust is safer than Haskell - FP Complete

We’re big fans of safety-focused languages at FP Complete. As our previous blog post comparing Rust and Haskell made clear, we think both of these are great languages. A recurring theme we have for both internal and client projects is: which language should we use? The general guidelines in that blog post apply:

Planet Haskell: FP Complete: When Rust is safer than Haskell - FP Complete

We’re big fans of safety-focused languages at FP Complete. As our previous blog post comparing Rust and Haskell made clear, we think both of these are great languages. A recurring theme we have for both internal and client projects is: which language should we use? The general guidelines in that blog post apply:

Colossal: A Traveling Exhibition of 100 Stunning Selections from the 2018 Wildlife Photographer of the Year

"The golden couple" by Marsel van Oosten, The Netherlands, Grand Title Winner 2018, Animal Portraits

“The golden couple,” Marsel van Oosten, The Netherlands, Grand Title Winner 2018, Animal Portraits

Last fall Dutch photographer Marsel van Oosten was the overall winner of the 54th annual Wildlife Photographer of the Year competition (previously) after being selected from over 45,000 submissions. His image, The Golden Couple, captures a pair of golden snub-nosed monkeys against a wooded backdrop. Their bright blue faces glow against the lush forest of China’s Qinling Mountains—the only habitat where the endangered primates are found in the wild.

In total there were 19 category winners from the tens of thousands who submitted images of wildlife and natural environments from all corners of the globe. Winning images included a wasp carrying a perfectly round segment of mud, two owls nestled snuggly in a pipe, and a leopard dreaming on a low branch. The winners, plus a selection of other entries from the competition, are currently in an exhibition which will travel to Canada, Spain, Australia, and Germany. The next stop for “Wildlife Photographer of the Year” is the Field Museum in Chicago, which opens March 22 and runs through January 2020. (via Block Club Chicago)

"Pipe owls," Arshdeep Singh, India, Winner 2018, 10 Years and Under

“Pipe owls,” Arshdeep Singh, India, Winner 2018, 10 Years and Under

"Hellbent," David Herasimtschuk, USA, Winner 2018, Behaviour: Amphibians and Reptiles

“Hellbent,” David Herasimtschuk, USA, Winner 2018, Behaviour: Amphibians and Reptiles

"Mud-rolling mud-dauber," Georgina Steytler, Australia, Winner 2018, Behaviour: Invertebrates

“Mud-rolling mud-dauber,” Georgina Steytler, Australia, Winner 2018, Behaviour: Invertebrates

"Night flight," Michael Patrick O’Neill, USA, Winner 2018, Under Water

“Night flight,” Michael Patrick O’Neill, USA, Winner 2018, Under Water

"Windsweep," Orlando Fernandez Miranda, Spain, Winner 2018, Earth’s Environments

“Windsweep,” Orlando Fernandez Miranda, Spain, Winner 2018, Earth’s Environments

"Mother defender," Javier Aznar González de Rueda, Spain, Winner 2018, Wildlife Photographer Portfolio Award

“Mother defender,” Javier Aznar González de Rueda, Spain, Winner 2018, Wildlife Photographer Portfolio Award

"Lounging leopard," by Skye Meaker, South Africa, Grand Title Winner 2018, 15-17 Years Old

“Lounging leopard,” Skye Meaker, South Africa, Grand Title Winner 2018, 15-17 Years Old

"Bed of seals," Cristobal Serrano, Spain, Winner 2018, Animals in their environment

“Bed of seals,” Cristobal Serrano, Spain, Winner 2018, Animals in their environment

 

explodingdog: Photo



Quiet Earth: John Wick: Chapter 3: Parabellum Trailer

John Wick is back in the first trailer for the franchise's third action packed installment. The first look at the film is as stylish and intense as you might expect.



Synopsis:
Super-Assassin John Wick (Keanu Reeves) is on the run after killing a member of the international assassin's guild, and with a $14 million price tag on his head - he is the target of hit men and women everywhere.



John Wick: Chapter 3 is directed by Chad Stahelski, who also directed the first two films.

Laurence Fishburne and Ian McShane will return to the franchise along with Halle Berry and an appearance by Jason Mantzoukas.


John Wick: Chapter 3 — Parabellum will hit theaters on May 17.


[Continued ...]

Tea Masters: Aged tea myth: Top Oriental Beauty from 2010

Fall 2010 top OB from Hsin Chu
Tea gardens are mostly asleep during winter. So, January is when tea farmers have the most time to spare taste tea with their customers. And since there are no fresh teas in January (except Dong Pian in Nantou), this is also a good time to explore their aged teas.

Before I introduce the nice fall 2010 top OB I've found in Hsin Chu county, let me give some warning about aged teas. A few weeks ago, a Taiwanese vendor was caught selling fake old tea by a Chinese customer. How did the customer find out? He used a detector of fluorescent chemicals and found some in the paper wrapping the tea. The paper was supposed to be as old as the tea, but these chemicals are much more recent. Apparently, this detector is quite cheap, but it's not something that most of us carry around in our pocket. Luckily, there are other ways to spot fakes. 
First brew
The most important tool is common sense. Does the story make sense or is it too good to be true? The first common sense observation is that time is money. Storing and keeping tea has a cost. An aged tea that costs less or about as much as new tea should be suspicious. The older the tea, the more suspicious. In the case of an old tea that costs about as much as a new tea, there are 2 possible explanations: 1. the tea is a fake, a new tea that has been processed to look and taste old (heavily roasted Oolong or wodui/cooked puerh). 2. the tea is old, but not very good, because it's a leftover from the past that has not aged well. (For instance, when Dong Ding Oolong became popular in the late 1970s, a lot of Wenshan Baozhong didn't sell anymore. Unfortunately, most of these Baozhongs were kept in cheap transparent plastic bags that didn't protect the leaves well.)

If the price of the tea is much more expensive than new tea, then you have to be even more careful! Again, some common sense:
- Packaging is easier to fake than tea. If you know what good aged tea looks and smells like, you should disregard the packaging and focus on the tea.
- Otherwise, ask yourself if the packaging is really as old as it claims to be. Try to research the company and see if they were really in business for this type of tea at the time claimed by the seller. Are there any mistakes or inconsistencies on the wrapper? I remember a fake where the telephone number indicated on the package had too many digits. Were teas packaged in such a manner in the past? Let's remember that before the second world war, tea was still a very luxurious product that had artfully decorated packaging. 
- High quality and expensive aged tea leaves should be clean and in excellent condition. With time, the scent is getting fainter. A strong scent is not normal for a very old tea.
In this case, 8 years isn't very old when it comes to an Oriental Beauty from a farmer in Hsin Chu who processes and roasts his OBs the traditional way. It's enough to compare the aged with the new to taste the difference and the improvement. The tea was wrapped in thick plastic foils that don't let sunshine in. And they were stored in a dry and clean place in his warehouse. Besides, there wasn't much left over in his inventory: about 3 kg only. This also makes sense, because farmers don't age big quantities of good tea on purpose.
Second brew
This aged Oriental Beauty Oolong has a very rich taste that stays strong, especially at the back of the throat. In the front of the mouth, it feels very smooth and oily. And it has a wonderful sweet candy fragrance! The roast flavors have mostly faded away, but I reckon that this OB still has many years of improvement ahead. Its taste hasn't reached its peak, yet. But it's already very pleasant to taste now or it could be a good choice if you wish to remember the year 2010 with a top OB.

BOOOOOOOM! – CREATE * INSPIRE * COMMUNITY * ART * DESIGN * MUSIC * FILM * PHOTO * PROJECTS: Artist Spotlight: Jesse Fillingham

A selection of recent work by artist Jesse Fillingham (previously featured here). See more images below.

Jesse Fillingham

 

 

Jesse Fillingham

 

 

Jesse Fillingham

 

 

Jesse Fillingham

 

 

Jesse Fillingham

 

 

Jesse Fillingham

 

 

Jesse Fillingham

 

 

Jesse Fillingham

 

 

Jesse Fillingham

 

 

Jesse Fillingham

 

 

Jesse Fillingham

 

 

Jesse Fillingham

 

 

Jesse Fillingham’s Website

Jesse Fillingham on Instagram

BOOOOOOOM! – CREATE * INSPIRE * COMMUNITY * ART * DESIGN * MUSIC * FILM * PHOTO * PROJECTS: Artist Spotlight: Gabrielle Teschner

Gabrielle Teschner

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner

 

 

Gabrielle Teschner’s Website

Gabrielle Teschner on Instagram

Penny Arcade: News Post: Audacious

Tycho: I push The Three-Body Problem on every-fuckin’-body.  People usually say it dips in the middle and finishes strong, but I can’t really agree with that - I think the story in general becomes something we’re not supposed to fully understand.  To explain why would be telling.  But the first volume is legitimately substantial and I went after it like a porterhouse. I pushed it on Gabriel for a specific reason: there is a concept in there that may be one most profound I’ve ever encountered in or out of fiction and I wanted to make sure he knew about it…

Quiet Earth: First Look at Dark Comedy HARPOON [Trailer]

The lost-at-sea trope usually plays out as a horror, thriller, drama or a combination of the three but comedy isn't, generally speaking, thrown into the mix.


Enter writer/director Rob Grant with his new feature Harpoon.


Starring Munro Chambers (Turbo Kid), Emily Tyra (Code Black), and Christopher Gray ("The Mist"), the movie is described as a "dark comedy about the rivalries, dark secrets, and sexual tension that amount when three best friends find themselves stranded on a yacht in the middle of the ocean under suspicious circumstances."


Though the first teaser trailer doesn't really provide much in the way of anything story-wise, I do love that it plays nothing like a comedy and therefore, I'm intrigued.


Harpoon will be making [Continued ...]

Quiet Earth: Cryptic Look at TIFF Winner CITIES OF LAST THINGS [Trailer]

Malaysian director Ho Wi-ding is no stranger to international acclaim. Two of his short films played at Cannes where he picked up both a Kodak discovery award and young critics award but his latest feature Cities of Last Things, is his most international production to date.


The movie spans decades over a man's life and focuses on three distinct moments which changed the course of his life but the triptych unfolds in reverse chronological order; opening in the future, then present and closing with the past.


The trailer is cryptic and if I hadn't read a synopsis, I doubt I would ever have figured out that the three men are all portraying the same man at different points in his life but the mystery adds to the movie's intrigue.


Cities of Last Things, [Continued ...]

Quiet Earth: Robert Pattinson Goes to Space in HIGH LIFE [Trailer]

As noted by Chris when the lovely poster for High Life premiered earlier this week, Claire Denis doesn't exactly deliver crowd pleaser material but if the teaser trailer and the newly minted trailer for her upcoming movie is any indication, she's definitely hitting all the right buttons for this viewer.


Set in the future, the movie stars Robert Pattinson as a guy struggling to survive on a spaceship with his baby daughter. How did he get there?


The new trailer suggests an apocalypse not unlike that imagined in Children of Men, with Pattinson, André Benjamin and Mia Goth being selected for a possible dead-end mission by Juliette Binoche.


Yes. Please.

[Continued ...]

Quiet Earth: Indie Comedy AMELIA’S 25th Lands January 18 [Trailer]

Global Digital Releasing will release the indie comedy, Amelia's 25th on digital this Friday, January 18. The star studded film features Danny Trejo, Margaret Cho, Robert Rodriguez, Jon Abrahams and Jennifer Tilly and featured Electra Avellan as Amelia.

The film centers around struggling actress Amelia Cruz as she attempts to celebrate her birthday in a town where 25 is the new 55.

Amelia struggles to not only navigate her way through life and the barrage of eccentric “Hollywood characters” who seem to be assaulting her senses at every turn, but - with the help of a myst [Continued ...]

Planet Haskell: Gabriel Gonzalez: Dhall - Year in review (2018-2019)

<html lang="" xml:lang="" xmlns="http://www.w3.org/1999/xhtml"><head> <meta charset="utf-8"/> <meta content="pandoc" name="generator"/> <meta content="width=device-width, initial-scale=1.0, user-scalable=yes" name="viewport"/> dhall-2018 <style type="text/css"> code{white-space: pre-wrap;} span.smallcaps{font-variant: small-caps;} span.underline{text-decoration: underline;} div.column{display: inline-block; vertical-align: top; width: 50%;} </style> <style type="text/css">a.sourceLine { display: inline-block; line-height: 1.25; } a.sourceLine { pointer-events: none; color: inherit; text-decoration: inherit; } a.sourceLine:empty { height: 1.2em; } .sourceCode { overflow: visible; } code.sourceCode { white-space: pre; position: relative; } div.sourceCode { margin: 1em 0; } pre.sourceCode { margin: 0; } @media screen { div.sourceCode { overflow: auto; } } @media print { code.sourceCode { white-space: pre-wrap; } a.sourceLine { text-indent: -1em; padding-left: 1em; } } pre.numberSource a.sourceLine { position: relative; left: -4em; } pre.numberSource a.sourceLine::before { content: attr(data-line-number); position: relative; left: -1em; text-align: right; vertical-align: baseline; border: none; pointer-events: all; display: inline-block; -webkit-touch-callout: none; -webkit-user-select: none; -khtml-user-select: none; -moz-user-select: none; -ms-user-select: none; user-select: none; padding: 0 4px; width: 4em; color: #aaaaaa; } pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; } div.sourceCode { } @media screen { a.sourceLine::before { text-decoration: underline; } } code span.al { color: #ff0000; font-weight: bold; } /* Alert */ code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */ code span.at { color: #7d9029; } /* Attribute */ code span.bn { color: #40a070; } /* BaseN */ code span.bu { } /* BuiltIn */ code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */ code span.ch { color: #4070a0; } /* Char */ code span.cn { color: #880000; } /* Constant */ code span.co { color: #60a0b0; font-style: italic; } /* Comment */ code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */ code span.do { color: #ba2121; font-style: italic; } /* Documentation */ code span.dt { color: #902000; } /* DataType */ code span.dv { color: #40a070; } /* DecVal */ code span.er { color: #ff0000; font-weight: bold; } /* Error */ code span.ex { } /* Extension */ code span.fl { color: #40a070; } /* Float */ code span.fu { color: #06287e; } /* Function */ code span.im { } /* Import */ code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */ code span.kw { color: #007020; font-weight: bold; } /* Keyword */ code span.op { color: #666666; } /* Operator */ code span.ot { color: #007020; } /* Other */ code span.pp { color: #bc7a00; } /* Preprocessor */ code span.sc { color: #4070a0; } /* SpecialChar */ code span.ss { color: #bb6688; } /* SpecialString */ code span.st { color: #4070a0; } /* String */ code span.va { color: #19177c; } /* Variable */ code span.vs { color: #4070a0; } /* VerbatimString */ code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */ </style> </head><body>

The Dhall configuration language is now two years old and this post will review progress in 2018 and the future direction of the language in 2019.

If you’re not familiar with Dhall, you might want to visit the official website for the language, which is the recommended starting point. This post assumes familiarity with the language.

Also, I want to use this post to advertise a short survey that you can take if you are interested in the language and would like to provide feedback:

Progress in 2018

This section will review the highlights of what we accomplished over the last year. These highlights are not exhaustive and I focus on improvements that might encourage people to revisit the language if they were on the fence a year ago.

If you’re already familiar with recent progress in the language and you are more interested in where the language is going then you can jump to the Future direction section.

New language bindings

Several contributors stepped up to the plate to begin three new actively-maintained language bindings to Dhall.

Of these three the Clojure bindings are the ones closest to completion:

The Clojure bindings are sufficiently close to completion that they currently get an official vote on proposed changes to the language standard, giving them an equal voice in the language evolution process.

This is a complete reimplementation of the language entirely in Clojure that allows you to marshal Dhall expressions, including Dhall functions, directly into Clojure:

The Clojure bindings pave the way for making the Dhall configuration language a first class citizen on the JVM.

Additionally, two other language bindings have gotten pretty far along:

These latter two language bindings haven’t announced yet as they are works in progress, but I still wanted to recognize their work so far.

Also, I want to mention that adding a conformance test suite (thanks to Fabrizio Ferrai) helped drive parallel implementations by providing implementors with a tangible measure of progress towards the goal of 100% standard coverage.

Haskell - Cabal

Thanks to the work of Oliver Charles you can generate .cabal files with Dhall by using dhall-to-cabal.

This part of the project’s README sold me on the motivation for doing so:

We can go beyond Cabal files. If Cabal is a domain specific language for building Haskell projects, what does a domain specific language for building Haskell web applications look like? Does the separate of library, executable, and test-suite make sense here? Maybe we’d rather:

… and have this take care of some other details.

When you think about this it makes perfect sense: Haskell programmers use Cabal/Hackage to package and distribute Haskell code, but then what do they use to package and distribute “Cabal code”? The answer is a language like Dhall that builds in its own code distribution mechanism instead of relying on a separate build tool. This closes the loop so that you don’t need to maintain a growing tower of build tools as your project expands.

I’m pretty sure Cabal was the first “heavy duty” configuration format tested with Dhall because this project prompted the first swell of feature requests related to interpreter performance and usability improvements for working with giant schemas.

Also, the dhall-to-cabal project includes the entire Cabal schema encoded as a Dhall type, which you can find here:

This comes in handy when you want a systematic listing of all Cabal configuration features. If you have the dhall interpreter installed you can also view the normal form of the schema in all its glory by running:

You can also migrate an existing project using cabal-to-dhall, a tool which converts a .cabal file to the equivalent .dhall file.

Eta

Javier Neira with the support of TypeLead added Dhall as a supported file format for configuring Eta packages by building on top of the dhall-to-cabal project.

That project has also produced work-in-progress Eta and Java bindings to Dhall bindings along the way by using Eta to compile the Haskell implementation of Dhall to the JVM. When those are complete you will have yet another option for using the Dhall configuration language on the JVM

If you are interested, you can follow the progress on those bindings via this GitHub issue:

Kubernetes

Dhall is commonly used for ops, and the first project to systematically integrate Dhall into a widely used ops tool is the dhall-kubernetes project, thanks to the work of Arian van Putten, Fabrizio Ferrai, and Thomas Scholtes.

I’ve never used Kubernetes, but everybody tells me that Kubernetes configurations are large, repetitive, and error-prone YAML files, which are the perfect use case for Dhall.

PureScript - Spago

The Spago project builds on top of psc-packages to assemble a PureScript package set to build using Dhall as the configuration format.

This tool takes advantage of Dhall’s import system so that the package set can be split over multiple files, which you can see in the top-level package set here:

… and users can easily import that and easily override packages locally without dealing with the headache of rebasing their local changes whenever the upstream package set changes.

Complete language standard

Last year I promised to upstream all features from the Haskell implementation into the language standard, since at the time a few import-related features were implementation-defined. This was a top priority because I didn’t want to treat other language bindings as second-class citizens.

This year we successfully standardized everything, meaning that there should no longer be any implementation-defined features. Additionally, all new functionality now begins with a change to the standard followed by a change to each implementation of the language, meaning that the Haskell implementation is no longer treated as a distinguished implementation.

Unsigned Natural literals

Earlier this year Greg Pfeil proposed to fix the obligatory + sign preceding Natural number literals, which bothered a lot of newcomers to the language. He proposed require a leading + for non-negative Integers instead of Natural numbers.

We knew this would be a highly breaking change, but we were all tired of the + signs which littered our code. The Natural type is much more natural to use (pun intended) than the Integer type, so why not optimize the syntax for Natural numbers?

So we made the change and now instead of writing an expression like:

… you instead write:

This change also improved code comprehension, because before this change an expression like this:

… could be misconstrued as adding f to various numbers, but after this change:

… the reader can more easily discern that f is being applied as a function to numeric arguments.

Type synonyms

Previously, users couldn’t create new types using let expressions and had to work around this limitation by using the import system to reuse types, like this:

Now users can define new types inline within the same file using an ordinary let expression:

Simpler Optional literals

Optional literals used to resemble lists, like this:

Now you can use Some and None instead, like this:

In particular, a present Optional literal no longer requires a type since Some can infer the type from the provided argument. This simplifies the common idiom of overriding an absent Optional value within a record of defaults:

The old list-like syntax is still supported but is deprecated and will be dropped. dhall lint will also automatically replace the old list-like Optional literals with their new Some/None equivalents.

Union constructors

Unions used to be one of the major pain points when using the language, due to having to specify all alternatives for a union literal, like this:

My first attempt to improve this introduced the constructors keyword which took a union type as an argument and returned a record of functions to create each constructor. This changed the above code example to:

However, this initial solution introduced two new problem:

  • A lot of constructors-related boilerplate at the beginning of Dhall files
  • Performance issues due to these large intermediate records of constructors

A follow-up change resolved both issues by overloading the . operator to also access constructor functions directly from a union type (as if it were already a record), like this:

This solved both the performance issue (by eliminating the need for an intermediate record of constructors) and eliminated the constructors keyword boilerplate. Also, this simplification plays nicely with the auto-complete support provided by the dhall repl since you can now auto-complete constructors using the . operator.

Import caching

In 2017 Dhall added support for semantic integrity checks, where you tag an import with a SHA256 hash of a standard binary encoding of an expression’s normal form. This integrity check protects against tampering by rejecting any expression with a different normal form, guaranteeing that the import would never change.

Several astute users pointed out that you could locally cache any import protected by such a check indefinitely. Even better, the SHA256 hash makes for a natural lookup key within that cache.

We standardized and implemented exactly that idea and now any import protected by an integrity check is permanently cached locally using the standard directory prescribed by the XDG Base Directory Specification.

For example, you can now import the entire Prelude as a package protected by an integrity check:

The first time you resolve the Prelude the import may take a bit (~7 seconds on my machine) to fetch the entire package, but the normal form is then stored locally in a 5 KB file:

… and then subsequent attempts to import the same Prelude resolve much more quickly (~80 ms on my machine).

This means that you can now cheaply import the entire Prelude in every file instead of separately importing each function that you use.

Alternative imports

The language now provides support for fallback imports using the ? operator if import resolution fails.

For example, you can use this feature to import an environment variable if present but gracefully fallback to another value if absent:

Or you can use the ? operator to provide alternative locations for obtaining an imported expression:

Multi-let expressions

You can now define multiple values within a single let expression instead of nesting let expressions. In other words, instead of this:

… you can now write this:

dhall lint will also automatically simplify any code using the old nested let style to use the new “multi-let” style.

Statically linked executables

The Haskell implementation of Dhall strives to be like the “Bash of typed functional programming”, but in order to do so the implementation needs to small, statically linked, and portable so that sysadmins don’t object to widely installing Dhall. In fact, if the executable satisfies those criteria then you don’t even need your sysadmin’s permission to try Dhall out within your own workspace.

Niklas Hambüchen made this possible through this through his general-purpose work on fully static Haskell executables built using Nix. Now Dhall’s continuous integration system produces small (< 3 MB) Linux executables that have no dependency footprint whatsoever.

Major performance improvements

The Haskell implementation of Dhall has made dramatic strides in performance improvements over the last year, motivated by projects with very large schemas, such as:

… as well as Formation’s internal use of Dhall which has led to them upstreaming many performance improvements to handle large Dhall programs.

Thanks to the work of Fintan Halpenny, Greg Pfeil, @quasicomputational, and others the Haskell implementation is between 1 to 3 orders of magnitude faster than it was a year ago, depending on the configuration file that you benchmark.

We’re also not done improving performance! We continue to improve as new projects continue to stretch the boundaries of what the language can do.

Type diffs

Large projects like these also led to usability improvements when working with gigantic types. The Haskell implementation now displays concise “type diffs” whenever you get a type mismatch so that you can quickly narrow down the problem no matter how much your configuration schema grows. This works no matter how deeply nested the error is.

For example, the following contrived example introduces four deeply nested errors in a gigantic schema (where the type is over 6000 lines long) and the error message still zeroes in on every error:

dhall repl

The Haskell implementation also added a REPL contributed by Oliver Charles that you can use to interactively interpret Dhall code, including sophisticated auto-completion support contributed by Basile Henry:

<script async="async" id="asciicast-219319" src="https://asciinema.org/a/219319.js"></script>

The REPL comes in handy when exploring large values or types, as illustrated by the dhall-nethack tutorial which uses the REPL:

dhall lint

The Haskell implementation also provides a useful dhall lint subcommand that you can use to not only format code but to also automatically improve the code in non-controversial ways.

For example, dhall lint will automatically remove unused let bindings and will simplify nested let expressions to instead take advantage of the newest multi-let feature.

dhall resolve --dot

Basile Henry also contributed support for visualizing the dependency tree of a Dhall expression like this:

The following tweet illustrates how to use this feature along with example output:

<script async="async" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>

dhall freeze

Thanks to Tobias Pflug you can also automatically take advantage of Dhall’s semantic integrity checks using the dhall freeze subcommand. This command fetches all imports within a Dhall expression and then automatically tags all of them with semantic integrity checks.

For example:

dhall-lang.org

A while ago, Neuman Vong advised me that if you want your open source project to take off, you need a logo, a website, and a live demo in the browser.

So I took that advice to heart and now Dhall has all three! You can try out the language live in your browser by visiting:

This allows people to “try before they buy” and the site links to several other useful resources, such as the …

Dhall wiki

The Dhall wiki contains several useful educational resources for learning the language. The organization of the wiki closely follows the guidelines from this handy post on writing documentation:

The main thing that is missing is to migrate the Haskell tutorial into a language-agnostic tutorial.

Twitter account

You can also now follow the official Twitter account for the language:

This account regularly posts news and tips about the language and ecosystem that you can use to stay abreast of recent progress.

Switch from IPFS to GitHub

Early on in the language history we used IPFS to distribute the Dhall Prelude, but due to reliability issues we’ve switched to using GitHub for hosting Dhall code.

There’s even a convenient link you can use to browse the Prelude:

Future direction

Phew! That was a lot to recap and I’m grateful to all the contributors who made that possible. Now we can review where the language is going.

First, I’m no longer benevolent dictator-for-life of the language. Each new reimplementation of the language gets a vote on the language standard and now that the Clojure implementation of Dhall is essentially complete they get an equal say on the evolution of the language. Similarly, once the PureScript bindings and Python bindings are close to complete they will also get a vote on the language standard, too.

However, I can still use this post to outline my opinion of where the language should go.

Crossing the Chasm

A colleague introduced me to the book Crossing the Chasm, which heavily influenced my approach to designing and marketing the language. The book was originally written for startups trying to gain mainstream adoption, but the book also strongly resonated with my experience doing open source evangelism (first for Haskell, and now Dhall).

The book explains that you need to first build a best-in-class solution for a narrowly-defined market. This in turn requires that you think carefully about what market you are trying to address and strategically allocate your limited resources to address that market.

So what “market” should Dhall try to address?

YAML

One of the clearest signals I’ve gotten from users is that Dhall is “the YAML killer”, for the following reasons:

  • Dhall solves many of the problems that pervade enterprise YAML configuration, including excessive repetition and templating errors

  • Dhall still provides many of the good parts of YAML, such as multi-line strings and comments, except with a sane standard

  • Dhall can be converted to YAML using a tiny statically linked executable, which provides a smooth migration path for “brownfield” deployments

Does that mean that Dhall is clearly the best-in-class solution for people currently using YAML?

Not quite. The key thing Dhall is missing for feature parity with YAML is a wide array of native language bindings for interpreting Dhall configuration files. Many people would prefer to use Dhall without having to invoke an external executable to convert their Dhall configuration file to YAML.

This is one of the reasons I’ve slowed down the rate of evolution of the standard so that many of the new language bindings have an opportunity to implement the full standard. Also, I expect that once more language bindings have votes on the standard evolution that will further stabilize the language since new features proposals will have a higher bar to clear.

That’s not to say that we will freeze the language, but instead we will focus on strategically spending our “complexity budget” on features that help displace YAML. If we spend our complexity budget on unrelated features then we will increase the difficulty of porting Dhall to new languages without addressing the initial use case that will help Dhall gain mainstream traction.

JSON integration

One of YAML’s features is that all JSON is also valid YAML, by definition. In fact, some people use YAML just for the fact that it supports both JSON and comments.

This suggests that Dhall, like YAML, should also natively support JSON in some way. Dhall’s issue tracker contains a few issues along these lines and the one I would most like to see completed this year is adding support for importing JSON files as Dhall expressions:

Editor support

Another thing Dhall is missing compared to YAML is widespread editor support. This is why another one of my goals for this year is to create a Dhall language server so that any editor that supports the language server protocol (basically all of them) would get Dhall support for free.

Ops

We can actually narrow down Dhall’s “market” further if we really want to be selective about what we work on. Dhall has also grown in popularity for simplifying ops-related configurations, providing several features that ops engineers care about:

  • Strong normalization

    Ops commonly suffers from the dilemma that too much repetition is error prone, but too much abstraction is also error prone if readers of the code can’t effectively audit what is going on. One of Dhall’s unique features is that all code is strongly normalizing, meaning that every expression can be reduced to an abstraction-free normal form. This is made possible by the fact that Dhall is not Turing-complete (another feature favored by Ops).

  • Absolute type safety

    Ops engineers care about reliability since they maintain the software that the rest of their company relies on and any outages can have devastating effects on both the product and the productivity of other engineers.

    This is one of the reason for the Ops flight from Turing-complete languages to inert configuration files like JSON/YAML because Turing-complete languages give you the tools to shoot yourself in the foot. However, Dhall strikes a balance between being programmable while still not being Turing-complete and having a type system with no escape hatches, so you’re incapable of shooting yourself in the foot.

  • Built-in support for importing code

    Another reason that Ops people hate programmable configuration files is that the programming language they pick typically comes with an external build tool for the language that adds one more layer to the tower of build tools that they have to maintain. Now they’ve just replaced one problem (a repetitive configuration file for their infrastructure) which a new problem (a repetitive configuration file for the build tool for the programming language they used to reduce the original repetition).

    Dhall solves this problem well by providing built-in language support for importing other code (similar to Bash and Nix, both also heavily used for Ops use cases). This means that Dhall provides a solid foundaton for their tower of automation because they don’t need to introduce another tool to support a growing Dhall codebase.

  • Dhall displaces YAML well

    YAML configuration files are incredibly common in Ops and “infrastructure as code”. Example tools that use a YAML configuration are:

    • Kubernetes
    • Docker Compose
    • Concourse
    • Ansible
    • Travis

    YAML is so common that Ops engineers sometimes half-jokingly refer to themselves as “YAML engineers”.

    As already mentioned above, Dhall provides a sane alternative to YAML.

We’ve already seen one Dhall integration for an Ops tool emerge last year with the dhall-kubernetes project and this year I hope we continue along those lines and add at least one more Ops-related integration.

I think the next promising integration is the dhall-terraform project which is still a work in progress that would benefit from contributions.

Funding

Finally, I would like to experiment with various ways to fund open source work on Dhall now that the language has a growing userbase. In particular, I’d like to fund:

  • additional language bindings
  • better editor support
  • adding CI support for statically linked Windows and OS X binaries
  • packaging Dhall for various software distributions (i.e. .rpm/.deb)

… and I’d like to provide some way to reward the work of people who contribute beyond just acknowledging their work in posts like this one.

That’s why one of the survey questions for this year asks for suggestions on what would be the most appropriate (non-proprietary) funding model for that sort of work.

Conclusion

Hopefully that gives people a sense of where I think the language is going. If you have any thoughts on the direction of the language this would be a good time to take the survey:

Like last year, I will follow-up a month from now with another post reviewing the feedback from the survey.

</body></html>

Penny Arcade: Comic: Audacious

New Comic: Audacious

Explosm.net: Comic for 2019.01.16

New Cyanide and Happiness Comic

Daniel Lemire's blog: Faster intersections between sorted arrays with shotgun

A common problem within databases and search engines is to compute the intersection between two sorted array. Typically one array is much smaller than the other one.

The conventional strategy is the “galloping intersection”. In effect, you go through the values in the small arrays and then do a binary search in the large array. A binary search is a simple but effective algorithm to search through a sorted array. Given a target, you compare with with the midpoint value. If your target is smaller than the midpoint value, you search in the first half of the array, otherwise you search in the second half. You can recurse through the array in this manner, cutting the search space in half each time. Thus the search time is logarithmic.

If the small array has M elements and the large array has N elements, then the complexity of a galloping search is O(M log N). In fact, you can be more precise: you never need more than M * log N + M comparisons.

Can you do better? You might.

Let me describe an improved strategy which I call “shotgun intersection”. It has been in production use for quite some time, through the CRoaring library, a C/C++ implementation of Roaring Bitmaps.

The idea is that galloping search implies multiple binary searches in sequence through basically the same array. Doing them consecutively might not be best. A binary search, when the large array is not in cache, is memory-bound: it waits for the memory subsystem to deliver the data. So you are constantly waiting. What if you tried to do something else while you wait. What about starting right away on the next binary search?

That is how a shotgun search works. You take, say, the first four values from the small array. You load the midpoint value from the large array, then you compare all of your four values against this midpoint value. If the target value is larger, you set a corresponding index so that the next search will hit the second half of the array. And so forth. In effect, shotgun search does many binary searches at once.

I make my Java code available, if you want a full implementation.

Does it help? It does. Sometimes it helps a lot. Let us intersect an array made of 32 integers with an array made of 100 million sorted integers. I use a cannonlake processor with Java 8.

1-way 1.3 microseconds
4-way 0.9 microseconds

Credit: Shotgun intersections are based on an idea and an initial implementation by Nathan Kurz. I’d like to thank Travis Downs for inspiring discussions.

OUR VALUED CUSTOMERS: The modern humorist's dream...

Planet Lisp: Zach Beane: Want to write Common Lisp for RavenPack? | R. Matthew Emerson

Want to write Common Lisp for RavenPack? | R. Matthew Emerson

OCaml Planet: Improving Tezos Storage

Running a Tezos node currently costs a lot of disk space, about 59 GB for the context database, the place where the node stores the states corresponding to every block in the blockchain, since the first one. Of course, this is going to decrease once garbage collection is integrated, i.e. removing very old information, that is not used and cannot change anymore (PR720 by Thomas Gazagnaire, Tarides, some early tests show a decrease to 14GB ,but with no performance evaluation). As a side note, this is different from pruning, i.e. transmitting only the last cycles for “light” nodes (PR663 by Thomas Blanc, OCamlPro). Anyway, as Tezos will be used more and more, contexts will keep growing, and we need to keep decreasing the space and performance cost of Tezos storage.

As one part of our activity at OCamlPro is to allow companies to deploy their own private Tezos networks, we decided to experiment with new storage layouts. We implemented two branches: our branch IronTez1 is based on a full LMDB database, as Tezos currently, but with optimized storage representation ; our branch IronTez2 is based on a mixed database, with both LMDB and file storage.

To test these branches, we started a node from scratch, and recorded all the accesses to the context database, to be able to replay it with our new experimental nodes. The node took about 12 hours to synchronize with the network, on which about 3 hours were used to write and read in the context database. We then replayed the trace, either only the writes or with both reads and writes.

Here are the results:

The mixed storage is the most interesting: it uses half the storage of a standard Tezos node !

Again, the mixed storage is the most efficient : even with reads and writes, IronTez2 is five time faster than the current Tezos storage.

Finally, here is a graph that shows the impact of the two attacks that happened in November 2018, and how it can be mitigated by storage improvement:

The graph shows that, using mixed storage, it is possible to restore the storage growth of Tezos to what it was before the attack !

Interestingly, although these experiments have been done on full traces, our branches are completely backward-compatible : they could be used on an already existing database, to store the new contexts in our optimized format, while keeping the old data in the ancient format.

Of course, there is still a lot of work to do, before this work is finished. We think that there are still more optimizations that are possible, and we need to test our branches on running nodes for some time to get confidence (TzScan might be the first tester !), but this is a very encouraging work for the future of Tezos !

 

 

 

Michael Geist: Why So Secret?: Government’s Communications Law Panel Plans to Keep Public Submissions Under Wraps for Months

The deadline for submissions to the government’s Broadcasting and Telecommunications Legislative Review Panel passed last week. I posted my submission yesterday, joined by several other organizations representing differing perspectives (CRTC, CBC, Friends of Canadian Broadcasting, Writers Guild of Canada, Internet Society Canada Chapter, CMCRP). However, public availability of submissions will apparently be the exception for the foreseeable future. The panel has rejected an open and transparent policy making process in which public submissions are publicly available, choosing instead to keep the submissions secret for months.

Parties submitting to the review process were advised that their submissions would be made public (“Written submissions will be made publicly available through the Panel website after the deadline.”). Yesterday, the panel’s website was updated to list the 124 organizations that have met with the panel behind closed doors (I met with the panel last fall) and to note that they have received 2,000 submissions. Yet despite the notification that submissions would be made public after the deadline, the panel has decided to make it long after the deadline with no intention of public posting until after the panel has released its initial report, which could come as late as June 30th.

The secrecy associated with a panel that already conducts most of its activities in secret is contrary to the open-by-default approach promised by the current government. Public availability of the submissions is typically the standard: submissions to the House of Commons committees conducting the copyright review are posted online, the CRTC posts the submissions it receives online, ISED posted responses to its public consultation on Copyright Board reform soon after the deadline closed, and Canadian Heritage posted submissions to its digital Cancon consultation within two weeks of the submission deadline. The notion of waiting months to post submissions to a public consultation would rightly be rejected by the CRTC or House committees as inconsistent with a transparent policy process.

Indeed, keeping the submissions secret for months benefits no one. Rather, it fuels concern about the secrecy of the panel process, it means that stakeholders are unable to assess and consider data and policy proposals from other stakeholders, and it leaves everyone wholly dependent on the panel for an accurate summation of thousands of submissions. What did Netflix say to the government? What about Bell or Rogers? What about the report commissioned by the CMPA from PricewaterhouseCoopers that purports to propose a new broadcast distribution system? How about submissions from individual Canadians concerned with communications policy?

For the responsible ministers – ISED Minister Navdeep Bains and Canadian Heritage Minister Pablo Rodriguez – this is a bad look that signals the panel is more interested in secrecy than sharing the substance behind what Canadians are saying about their communications law. I have filed an Access to Information request for the submissions (and would be happy to update this post with links to other publicly posted submissions), but ATIPs should not be required for public submissions. The panel should reconsider its approach and post all submissions as soon as possible.

[Update: Submissions also posted by:

The post Why So Secret?: Government’s Communications Law Panel Plans to Keep Public Submissions Under Wraps for Months appeared first on Michael Geist.

bit-player: Glauber’s dynamics

Roy J. Glauber, Harvard physics professor for 65 years, longtime Keeper of the Broom at the annual Ig Nobel ceremony, and winner of a non-Ig Nobel, has died at age 93. Glauber is known for his work in quantum optics; roughly speaking, he developed a mathematical theory of the laser at about the same time that device was invented, circa 1960. His two main papers on the subject, published in Physical Review in 1963, did not meet with instant acclaim; the Nobel committee’s recognition of their worth came more than 40 years later, in 2005. A third paper from 1963, titled “Time-dependent statistics of the Ising model,” also had a delayed impact. It is the basis of a modeling algorithm now called Glauber dynamics, which is well known in the cloistered community of statistical mechanics but deserves wider recognition.

Before digging into the dynamics, however, let us pause for a few words about the man himself, drawn largely from the obituaries in the New York Times and the Harvard Crimson.

Glauber was a member of the first class to graduate from the Bronx High School of Science, in 1941. From there he went to Harvard, but left in his sophomore year, at age 18, to work in the theory division at Los Alamos, where he helped calculate the critical mass of fissile material needed for a bomb. After the war he finished his degree at Harvard and went on to complete a PhD under Julian Schwinger. After a few brief adventures in Princeton and Pasadena, he was back at Harvard in 1952 and never left. A poignant aspect of his life is mentioned briefly in a 2009 interview, where Glauber discusses the challenge of sustaining an academic career while raising two children as a single parent.


Here’s a glimpse of Glauber dynamics in action. Click the Go button, then try fiddling with the slider.

3.00

In the computer program that drives this animation, the slider controls a variable representing temperature. At high temperature (slide the control all the way to the right), you’ll see a roiling, seething mass of colored squares, switching rapidly and randomly between light and dark shades. There are no large-scale or long-lived structures. Occasionally the end point is not a monochromatic field. Instead the panel is divided into broad stripes—horizontal, vertical, or diagonal. This is an artifact of the finite size of the lattice and the use of wraparound boundary conditions. On an infinite lattice, the stripes would not occur.At low temperature (slide to the left), the tableau congeals into a few writhing blobs of contrasting color. Then the minority blobs are likely to evaporate, and you’ll be left with an unchanging, monochromatic panel. Between these extremes there’s some interesting behavior. Adjust the slider to a temperature near 2.27 and you can expect to see persistent fluctuations at all possible scales, from isolated individual blocks to patterns that span the entire array.

What we’re looking at here is a simulation of a model of a ferromagnet—the kind of magnet that sticks to the refrigerator. The model was introduced almost 100 years ago by Wilhelm Lenz and his student Ernst Ising. They were trying to understand the thermal behavior of ferromagnetic materials such as iron. If you heat a block of magnetized iron above a certain temperature, called the Curie point, it loses all traces of magnetization. Slow cooling below the Curie point allows it to spontaneously magnetize again, perhaps with the poles in a different orientation. The onset of ferromagnetism at the Curie point is an abrupt phase transition.

Lenz and Ising created a stripped-down model of a ferromagnet. In the two-dimensional version shown here, each of the small squares represents the spin vector of an unpaired electron in an iron atom. The vector can point in either of two directions, conventionally called up and down, which for graphic convenience are represented by two contrasting colors. There are \(100 \times 100 = 10{,}000\) spins in the array. This would be a minute sample of a real ferromagnet. On the other hand, the system has \(2^{10{,}000}\) possible states—quite an enormous number.

The essence of ferromagnetism is that adjacent spins “prefer” to point in the same direction. To put that more formally: The energy of neighboring spins is lower when they are parallel, rather than antiparallel. four possible orientations of two spins (up up, up down, down up, down down), with their corresponding energiesFor the array as a whole, the energy is minimized if all the spins point the same way, either up or down. Each spin contributes a tiny magnetic moment. When the spins are parallel, all the moments add up and the system is fully magnetized.

If energy were the only consideration, the Ising model would always settle into a magnetized configuration, but there is a countervailing influence: Heat tends to randomize the spin directions. At infinite temperature, thermal fluctuations completely overwhelm the spins’ tendency to align, and all states are equally likely. Because the vast majority of those \(2^{10{,}000}\) configurations have nearly equal numbers of up and down spins, the magnetization is negligible. At zero temperature, nothing prevents the system from condensing into the fully magnetized state. The interval between these limits is a battleground where energy and entropy contend for supremacy. Clearly, there must be a transition of some kind. For Lenz and Ising in the 1920s, the crucial question was whether the transition comes at a sharply defined critical temperature, as it does in real ferromagnets. A more gradual progression from one regime to the other would signal the model’s failure to capture important aspects of ferromagnet physics.

In his doctoral dissertation Ising investigated the one-dimensional version of the model—a chain or ring of spins, each one holding hands with its two nearest neighbors. The result was a disappointment: He found no abrupt phase transition. And he speculated that the negative result would also hold in higher dimensions. The Ising model seemed to be dead on arrival.

It was revived a decade later by Rudolf Peierls, who gave suggestive evidence for a sharp transition in the two-dimensional lattice. Then in 1944 Lars Onsager “solved” the two-dimensional model, showing that the phase transition does exist. The phase diagram looks like this:

Graph of the gagnetization v temperature phase diagram for the Ising model.

As the system cools, the salt-and-pepper chaos of infinite temperature evolves into a structure with larger blobs of color, but the up and down spins remain balanced on average (implying zero magnetization) down to the critical temperature \(T_C\). At that point there is a sudden bifurcation, and the system will follow one branch or the other to full magnetization at zero temperature.


If a model is classified as solved, is there anything more to say about it? In this case, I believe the answer is yes. The solution to the two-dimensional Ising model gives us a prescription for calculating the probability of seeing any given configuration at any given temperature. That’s a major accomplishment, and yet it leaves much of the model’s behavior unspecified. The solution defines the probability distribution at equilibrium—after the system has had time to settle into a statistically stable configuration. It doesn’t tell us anything about how the lattice of spins reaches that equilibrium when it starts from an arbitrary initial state, or how the system evolves when the temperature changes rapidly.

It’s not just the solution to the model that has a few vague spots. When you look at the finer details of how spins interact, the model itself leaves much to the imagination. When a spin reacts to the influence of its nearest neighbors, and those neighbors are also reacting to one another, does everything happen all at once? Suppose two antiparallel spins both decide to flip at the same time; they will be left in a configuration that is still antiparallel. It’s hard to see how they’ll escape repeating the same dance over and over, like people who meet head-on in a corridor and keep making mirror-image evasive maneuvers. This kind of standoff can be avoided if the spins act sequentially rather than simultaneously. But if they take turns, how do they decide who goes first?

Within the intellectual traditions of physics and mathematics, these questions can be dismissed as foolish or misguided. After all, when we look at the procession of the planets orbiting the sun, or at the colliding molecules in a gas, we don’t ask who takes the first step; the bodies are all in continuous and simultaneous motion. Newton gave us a tool, calculus, for understanding such situations. If you make the steps small enough, you don’t have to worry so much about the sequence of marching orders.

However, if you want to write a computer program simulating a ferromagnet (or simulating planetary motions, for that matter), questions of sequence and synchrony cannot be swept aside. With conventional computer hardware, “let everything happen at once” is not an option. The program must consider each spin, one at a time, survey the surrounding neighborhood, apply an update rule that’s based on both the state of the neighbors and the temperature, and then decide whether or not to flip. Thus the program must choose a sequence in which to visit the lattice sites, as well as a sequence in which to visit the neighbors of each site, and those choices can make a difference in the outcome of the simulation. So can other details of implementation. Do we look at all the sites, calculate their new spin states, and then update all those that need to be flipped? Or do we update each spin as we go along, so that spins later in the sequence will see an array already modified by earlier actions? The original definition of the Ising model is silent on such matters, but the programmer must make a commitment one way or another.

This is where Glauber dynamics enters the story. Glauber presented a version of the Ising model that’s somewhat more explicit about how spins interact with one another and with the “heat bath” that represents the influence of temperature. It’s a theory of Ising dynamics because he describes the spin system not just at equilibrium but also during transitional stages. I don’t know if Glauber was the first to offer an account of Ising dynamics, but the notion was certainly not commonplace in 1963.


There’s no evidence Glauber was thinking of his method as an algorithm suitable for computer implementation. The subject of simulation doesn’t come up in his 1963 paper, where his primary aim is to find analytic expressions for the distribution of up and down spins as a function of time. (He did this only for the one-dimensional model.) Nevertheless, Glauber dynamics offers an elegant approach to programming an interactive version of the Ising model. Assume we have a lattice of \(N\) spins. Each spin \(\sigma\) is indexed by its coordinates \(x, y\) and takes on one of the two values \(+1\) and \(-1\). Thus flipping a spin is a matter of multiplying \(\sigma\) by \(-1\). The algorithm for a updating the lattice looks like this:

Repeat \(N\) times:

  1. Choose a spin \(\sigma_{x, y}\) at random.
  2. Sum the values of the four neighboring spins, \(S = \sigma_{x+1, y} + \sigma_{x-1, y} + \sigma_{x, y+1} + \sigma_{x, y-1}\). The possible values of \(S\) are \(\{-4, -2, 0, +2, +4\}\).
  3. Calculate \(\Delta E = 2 \, \sigma_{x, y} \, S\), the change in interaction energy if \(\sigma_{x, y}\) were to flip.
  4. If \(\Delta E \lt 0\), set \(\sigma_{x, y} = -\sigma_{x, y}\).
  5. Otherwise, set \(\sigma_{x, y} = -\sigma_{x, y}\) with probability \(\exp(-\Delta E/T)\), where \(T\) is the temperature.

Display the updated lattice.

Step 4 says: If flipping a spin will reduce the overall energy of the system, flip it. Step 5 says: Even if flipping a spin raises the energy, go ahead and flip it in a randomly selected fraction of the cases. The probability of such spin flips is the Boltzmann factor \(\exp(-\Delta E/T)\). This quantity goes to \(0\) as the temperature \(T\) falls to \(0\), so that energetically unfavorable flips are unlikely in a cold lattice. The probability approaches \(1\) as \(T\) goes to infinity, which is why the model is such a seething mass of fluctuations at high temperature.

(If you’d like to take a look at real code rather than pseudocode—namely the JavaScript program running the simulation above—it’s on GitHub.)

Glauber dynamics belongs to a family of methods called Markov chain Monte Carlo algorithms (MCMC). The idea of Markov chains was an innovation in probability theory in the early years of the 20th century, extending classical probability to situations where the the next event depends on the current state of the system. Monte Carlo algorithms emerged at post-war Los Alamos, not long after Glauber left there to resume his undergraduate curriculum. He clearly kept up with the work of Stanislaw Ulam and other former colleagues in the Manhattan Project.

Within the MCMC family, the distinctive feature of Glauber dynamics is choosing spins at random. The obvious alternative is to march methodically through the lattice by columns and rows, examining every spin in turn. Blinking checkerboardThat procedure can certainly be made to work, but it requires care in implementation. At low temperature the Ising process is very nearly deterministic, since unfavorable flips are extremely rare. When you combine a deterministic flip rule with a deterministic path through the lattice, it’s easy to get trapped in recurrent patterns. For example, a subtle bug yields the same configuration of spins on every step, shifted left by a single lattice site, so that the pattern seems to slide across the screen. Another spectacular failure gives rise to a blinking checkerboard, where every spin is surrounded by four opposite spins and flips on every time step. Avoiding these errors requires much fussy attention to algorithmic details. (My personal experience is that the first attempt is never right.)

Choosing spins by throwing random darts at the lattice turns out to be less susceptible to clumsy mistakes. Yet, at first glance, the random procedure seems to have hazards of its own. In particular, choosing 10,000 spins at random from a lattice of 10,000 sites does not guarantee that every site will be visited once. On the contrary, a few sites will be sampled six or seven times, and you can expect that 3,679 sites (that’s \(1/e \times 10{,}000)\) will not be visited at all. Doesn’t that bias distort the outcome of the simulation? No, it doesn’t. After many iterations, all the sites will get equal attention.

The nasty bit in all Ising simulation algorithms is updating pairs of adjacent sites, where each spin is the neighbor of the other. Which one goes first, or do you try to handle them simultaneously? The column-and-row ordering maximizes exposure to this problem: Every spin is a member of such a pair. Other sequential algorithms—for example, visiting all the black squares of a checkerboard followed by all the white squares—avoid these confrontations altogether, never considering two adjacent spins in succession. Glauber dynamics is the Goldilocks solution. Pairs of adjacent spins do turn up as successive elements in the random sequence, but they are rare events. Decisions about how to handle them have no discernible influence on the outcome.


Years ago, I had several opportunities to meet Roy Glauber. Regrettably, I failed to take advantage of them. Glauber’s office at Harvard was in the Lyman Laboratory of Physics, a small isthmus building connecting two larger halls. In the 1970s I was a frequent visitor there, pestering people to write articles for Scientific American. It was fertile territory; for a few years, the magazine found more authors per square meter in Lyman Lab than anywhere else in the world. But I never knocked on Glauber’s door. Perhaps it’s just as well. I was not yet equipped to appreciate what he had to say.

Now I can let him have the last word. This is from the introduction to the paper that introduced Glauber dynamics:

If the mathematical problems of equilibrium statistical mechanics are great, they are at least relatively well-defined. The situation is quite otherwise in dealing with systems which undergo large-scale changes with time. The principles of nonequilibrium statistical mechanics remain in largest measure unformulated. While this lack persists, it may be useful to have in hand whatever precise statements can be made about the time-dependent hehavior of statistical systems, however simple they may be.

CreativeApplications.Net: Alias – A teachable “parasite” for your smart assistant

Alias – A teachable “parasite” for your smart assistant
Created by Bjørn Karmann and Tore Knudsen, Alias is a teachable “parasite” that is designed to give users more control over their smart assistants, both when it comes to customisation and privacy.

OCaml Weekly News: OCaml Weekly News, 15 Jan 2019

  1. Dune 1.6.3
  2. Working with bit-level data

Penny Arcade: News Post: My new D&amp;D Campaign

Gabe: It’s been about eight years since I ran a home campaign but I’ve decided to start another one. My players are going to be my wife Kara along with my friends Amy the Falcone and Levin Sadsad. I asked them if they wanted to play straight up D&D or if I could futz with it a little bit. They all agreed that they were up for some futzing and so I went to work building out a little custom campaign setting. So we will be playing D&D 5e but it will be set in the world of Brightgrave. Here’s what the players know so far. BRIGHTGRAVE Not much is known about the events that took place…

Explosm.net: Comic for 2019.01.15

New Cyanide and Happiness Comic

things magazine: Click track

The Tom Thomson Catalogue Raisonné, worth a browse for those who love their brushstrokes / What things flopped because they arrived at exactly the wrong time? / Distant Worlds 2, a virtual expedition to Beagle Point, 65,000 light years from … Continue reading

Michael Geist: All About the Internet: My Submission to the Broadcasting and Telecommunications Legislative Review Panel on the Future of Canadian Communications Law

The deadline for submissions to the Broadcasting and Telecommunications Legislative Review Panel closed on Friday with a handful of organizations such as the CRTC, CBC, and Friends of Canadian Broadcasting posting their submissions online. My full submission can be found here.  I argue that Canada’s regulatory approach should be guided by a single, core principle: communications policy, whether telecommunications or broadcasting, is now – or will soon become –  Internet policy. This emerging communications world is mediated through the Internet and communications regulatory choices are therefore fundamentally about regulating or governing the Internet. My submission identifies four goals that should guide Canadian communications law and regulation:

1.    Universal, affordable access to the network
2.    Level regulatory playing field
3.    Regulatory humility
4.    Fostering competitiveness in the communications sector

The executive summary on each of the four issue is posted below, followed by a list of 23 recommendations contained in the submission. In the coming days, I’ll have posts that unpack some of the key issues.

Executive Summary

Universal Affordable Access to the Network

In a world in which Internet access is the gateway to communications, culture, commerce, education, and community participation, the single most important policy goal of communications legislation is universal, affordable Internet access. This submission discusses five issues central to a developing a legislative framework premised on universal, affordable Internet access.

The challenge associated with the Canadian access issue is well-known: a large geographic footprint that makes providing access to some rural and remote communities challenging, competitive shortcomings that have left Canada with an intractable digital divide that results in adoption rates that lag behind access rates, and regulatory and government policies that have consistently failed to achieve the goal of universal, affordable access. The panel should deliver a clear statement in support of universal access at the more ambitious speeds. Moreover, it should recommend mandated broadband obligations in support of affordability, limits on data caps, prioritization of adoption rates to close the digital divide, and establish a clear timeline for all stakeholders on achieving the universal, affordable access goal.

New policy measures to enhance Canadian wireless competition are also needed. The panel should recommend policies that support greater wireless competition. These include encouraging foreign investment, continuing to set aside spectrum for new entrants and smaller providers, and opposition to further marketplace consolidation on competition grounds. Moreover, a mandated MVNO policy, which would support nimble, low-cost competitors leading to more innovative pricing and services, is long overdue.

The panel should also recommend the implementation of a new policy direction that requires all decisions be assessed through the lens of their impact on affordable access. Policies that are likely to increase consumer costs on Internet access should be reviewed with goal of amendment to develop cost-neutral alternatives. Moreover, the Commission should be required to develop an Internet access impact assessment, akin to a privacy impact assessment, to fully explain the implications of decisions on the foundational goal of affordable access.

The panel should recommend binding consumer protection rules, truth in advertising requirements, safeguards against unfair charges or above-market roaming fees, mandated service disclosure requirements, and clear options for redress for aggrieved consumers. Transparency and a complaints mechanism are important, but the communications law framework should strengthen current consumer protections.

Ensuring that the affordability issue is an integral part of the policy process should not be limited to the policy direction. Effective and fair regulatory and consultative processes depend upon a myriad of perspectives and voices, particularly for those groups who often find themselves under-resourced and under-represented. The panel should recommend the establishment of consistent, stable funding for public interest participation in regulatory and policy proceedings. This should include the possibility of multi-year support for established organizations and proceeding-specific support for all eligible groups.

Level Playing Field for Policy: Equality Of Opportunity And Access

The Internet offers remarkable opportunities for all Canadians to create, communicate, and engage in civic activities. While Canadian communications law has long sought to identify “policy objectives” for the broadcast or telecom system, the Internet is far bigger than either of those systems. The panel should instead ensure that Canadian law offers a level playing field from a policy perspective thereby prioritizing equality of opportunity and access for all.

Despite the political affirmations of support for net neutrality, the panel should recommend an unequivocal legislative direction to support and enforce net neutrality.

A level playing field for policy should also include measures to enhance competition in the provision of access services. Given the enormous advantages wielded by incumbent providers, Canada suffers from insufficient competition, which leads to high prices, low usage rates relative to other developed countries, and affordability concerns for consumers with low household income. The panel should recommend a mandated MVNO system and an enhanced third-party access model that seeks to eliminate delays, establishes benchmarks for access, and features independent reviews of reported problems in facilitating consumer access.

The emergence of new online audio and video services has sparked considerable debate over whether or how to regulate services that typically fall outside the current regulatory framework. The panel should ensure that like is treated as like, with sufficient differentiation to treat similar services in an equivalent manner for regulatory purposes.

Humility In Regulation: Recognize The Limits Of Communications Law And Regulation

The Internet is not the equivalent of the broadcasting system and efforts to cast it as such for regulatory purposes are enormously problematic. Indeed, this submission argues that the panel should recognize the importance of regulatory humility as a fundamental principle, guided by the view that communications law should not be used as a regulatory mechanism when other, more appropriate regulatory or legal tools are available nor should it be relied upon as a critical funding mechanism to support other policy objectives.

Humility in regulation touches on numerous issues, but this submission is limited to two types: (i) overlapping regulation that engages issues such as freedom of expression, copyright and privacy; and (ii) cross-subsidization, in which communications law is used to subsidize policy goals in other sectors such as the sustainability of the media and support for Canadian cultural production.

There is unquestionably a need for laws that address expression online that are consistent with the Canadian Charter of Rights and Freedoms. However, speech regulation through communications law licensing or other mandated requirements cannot be easily justified when there are other, more appropriate and less invasive avenues to address online expression. The panel should therefore recommend that communications law defers to generally applicable laws to the maximum extent possible when addressing expression online.

The panel should reject any effort to revive a site blocking system within Canadian communications law, leaving the issue to copyright policy makers. Consistent with the benefits of reducing overlapping regulation, the panel should also recommend the elimination of privacy rules within Canadian communications law accompanied by a more robust, enforceable PIPEDA that could be used to address privacy safeguards within the sector.

Cultural cross-subsidization has been a hallmark of the Canadian communications system, with mandated contributions from broadcasters and broadcast distributors (BDUs) used to support the creation of Cancon. Rather than expanding the cross-subsidization approach, however, the panel should recommend its gradual elimination. This does not mean that there should not be public support for Cancon. Cancon support remains an important ingredient in a vibrant Canadian cultural sector. Rather, public support such as grants, tax benefits, and other measures should come from general revenues as a matter of public policy, not through cross-subsidization.

The panel should recommend emulating the government’s support for the media sector, which rightly adopts the position that if the media needs public support and the government believes it is in the public interest to do so, funding should come from general revenues as part of broader government policy, not through cross-subsidization and a myriad of levies that run counter to other policy goals such as affordable Internet access and marketplace innovation.

With respect to film and television production, the panel should recommend implementing a level playing field with regard to taxation by supporting the application of sales taxes and general income taxes to Internet services. These taxes of general application should be applied to all businesses doing business in Canada with the resulting revenues available to help fund support programs for Canadian content creation. While supporting the application of general taxes, the panel should reject the implementation of new taxes or mandated contributions on OTT services and Internet providers, the latter of which would increase the costs of access counter the foundational policy of affordable, universal access. Extending the cross-subsidization model to the Internet raises significant concerns associated with both over-regulation and increased Internet access costs.

Fostering Competitiveness in the Communications Sector

When the Broadcasting Act was crafted, broadcasters occupied a privileged position, since the creation of video was expensive and the spectrum needed to distribute it scarce. As a result, the government established a licensing system complete with content requirements and cultural contributions designed to further a myriad of policy goals. Yet among the more than 40 policy goals found in the current law, the word “competition” does not appear once. The absence of competition may have made sense when there was little of it, but in today’s world of abundance featuring a seemingly unlimited array of content and distribution possibilities, fostering competition among broadcasters and BDUs is essential to long-term marketplace success.

The panel should recommend several reforms that would help solidify the competitiveness of the sector. These include the removal of foreign ownership restrictions, enhancing consumer choice, the gradual elimination of simultaneous substitution, and limitations on the CBC’s acceptance of digital advertising to decrease overlap with the private sector advertising-based models. The submission does not recommend the elimination of mandated contributions by broadcasters and BDUs given the ongoing benefits those sectors enjoy, though it recognizes that the impact of those contributions is likely to diminish over time.

Summary of Recommendations

Universal, Affordable Internet Access

1.    The panel should deliver a clear statement in support of universal access at the more ambitious speeds.

2.    The panel should recommend mandated broadband obligations in support of affordability, limits on data caps, prioritization of adoption rates to close the digital divide, and establish a clear timeline for all stakeholders on achieving the universal, affordable access goal.

3.    The panel should recommend policies that support greater wireless competition. These include encouraging foreign investment, continuing to set aside spectrum for new entrants and smaller providers, and oppose further marketplace consolidation on competition grounds. Moreover, a mandated MVNO policy, which would support nimble, low-cost competitors leading to more innovative pricing and services, is long overdue.

4.    The panel should recommend the implementation of a new policy direction that require all decisions to be assessed through the lens of their impact on affordable access.

5.    The panel should recommend binding consumer protection rules, truth in advertising requirements, safeguards against unfair charges or above-market roaming fees, mandated service disclosure requirements, and clear options for redress for aggrieved consumers.

6.    The panel should recommend the establishment of consistent, stable funding for public interest participation in regulatory and policy proceedings

Level Playing Field for Policy: Equality Of Opportunity And Access

7.    The panel should ensure that Canadian law offers a level playing field from a policy perspective.

8.    The panel should recommend an unequivocal legislative direction to support and enforce net neutrality.

9.    The panel should recommend an enhanced third-party access model that seeks to eliminate delays, establishes benchmarks for access, and features independent reviews of reported problems in facilitating consumer access.

10.    The panel should ensure that like is treated as like, with sufficient differentiation to treat similar services in an equivalent manner for regulatory purposes.

Humility In Regulation: Recognize The Limits Of Communications Law And Regulation

11.    The panel should recognize the importance of regulatory humility as a fundamental principle, guided by the view that communications law should not be used as a regulatory mechanism when other, more appropriate regulatory or legal tools are available nor should it be relied upon as a critical funding mechanism to support other policy objectives.

12.    The panel should recommend that communications law defers to generally applicable laws to the maximum extent possible when addressing expression online.

13.    The panel should reject any effort to revive a site blocking system within Canadian communications law, leaving the issue to copyright policy makers.

14.    The panel should recommend the elimination of privacy rules within Canadian communications law accompanied by a more robust, enforceable PIPEDA that could be used to address privacy safeguards within the sector.

15.    Rather than expanding the cross-subsidization approach, the panel should recommend its gradual elimination.

16.    The panel should recommend emulating the government’s approach to assistance to the media sector as funding should come from general revenues as part of broader government policy, not through cross-subsidization and a myriad of levies that run counter to other policy goals such as affordable Internet access and marketplace innovation.

17.    The panel should recommend implementing a level playing field with regard to taxation by supporting the application of sales taxes and general income taxes to Internet services.

18.    The panel should reject the implementation of new taxes or mandated contributions on OTT services and Internet providers

Fostering Competitiveness in the Communications Sector

19.    The panel should recommend the elimination of foreign ownership restrictions in the licensed broadcasting sector.

20.    The panel should recommend establishing more robust, mandated options to enhance consumer choice and drive increased competitiveness in the sector.

21.    The panel should recommend the gradual elimination of simultaneous substitution policies.

22.    The panel should recommend requiring the public broadcaster to adopt an ad-free approach to its online news presence.

23.    The panel should recommend an even bigger goal for the CBC to capture the public’s imagination. That could include requiring the CBC to open its content for public reuse or embarking on a comprehensive digitization initiative that provides the foundation for a national digital library.

The post All About the Internet: My Submission to the Broadcasting and Telecommunications Legislative Review Panel on the Future of Canadian Communications Law appeared first on Michael Geist.

Tea Masters: Puerh nostalgique du début des années 1970


 Voici mon Chaxi de ce samedi, à six mille lieues des manifestations en France. Mais bizarrement, et malgré la chance que j'ai de résider à Taiwan, je ne peux m'empêcher de me sentir triste et préoccupé par la mauvaise passe que traverse mon pays. Même marié à une Taiwanaise et résident local depuis 22 ans avec mes enfants scolarisés dans le système chinois et ma passion pour le thé et la culture chinoise, je continue de me sentir pleinement français.

Le thé et l'histoire de la Chine contiennent de nombreuses leçons pour la crise en France, mais ce blog n'a pas pour vocation d'être politique et clivant. Au contraire, j'aimerais pouvoir créer du consensus et du plaisir partagé avec mes expériences et le plaisir de mes Chaxi. L'idée de celui-ci est de retourner au début des années 1970, avant la première crise économique qui mit fin à la croissance des 30 glorieuses. C'est pourquoi, je choisis ce puerh cru en vrac. Et pour l'infuser, ma théière zisha décoré d'un lion. Car c'était l'époque où de grands hommes (de Gaulle, Pompidou) présidaient au pays. Ils servaient la nation avec la férocité du roi de la jungle. Sans déficits, sans dette, sans chômage.
 Qu'il est doux et puissant, pur et riche ce puerh et le souvenir de cette époque! Il a des odeurs de vieux bois noble, une douceur qui approche celle d'un miel de forêt sauvage foncé ou de mélasse, de sucre brun. Au goût, c'est le paradoxe d'une sensation d'extrême pureté et d'une énergie sous-jacente qui s'étend à travers tout le corps. Grand bien-être!
Vision d'un beau coucher de soleil sur une montagne (un pays?) lointaine. Je fais un voeu que la nuit qui vient sera de courte durée et que nous assisterons bientôt à un magnifique lever de soleil! Et si j'ai l'occasion de débattre et de donner des idées de ce qui marche à Taiwan, je le ferai.

Explosm.net: Comic for 2019.01.14

New Cyanide and Happiness Comic

Planet Lisp: Daniel Vedder: Silicon Valley Syndrome

“Silicon Valley Syndrome” is the name I give to a wide-spread myth that is frequently found in affluent, tech-savvy circles. It is the belief that “Every social problem can be solved if you just throw enough technology at it”. This belief lies at the heart of many, many attempts to make the world a better place. Their proponents will say things like: “We can save democracy by combating fake news with algorithms”, or “We can solve Third World hunger using satellite imagery”, or “We can improve education in poor areas by giving every kid an iPad”. These are all laudable attempts, and yet their fundamental assumption is all too often sadly misguided. Why is that?

* * *

The crux of the matter is that social problems are never just technical, but always at least in part human. Societies are made up of humans, and anybody who ignores this human aspect is doomed to (at least partial) failure.

Let us start with a personal example. All students and freelancers know the problem of procrastination. There are thousands of tools devoted to helping poor souls better manage their time (most of them based on some variation of the humble to-do list). But no to-do list, no matter how technologically advanced, has ever forced anybody to do anything. In the end, it is still the individual's choice whether or not to follow his or her plan, or to procrastinate. To-do lists can help keep us organised, but they cannot of themselves change our behaviour. Thus, they do not solve the underlying problem. In the end, procrastination remains a human issue.

We see the same pattern when we turn our attention to the higher level of social problems. One area that seems to be particularly susceptible to the Silicon Valley Syndrome is development aid. Growing up in Africa, I have had a fair bit of exposure to this, so let us take a closer look at some examples from this field.

Sometime in the last two years, friends of mine in Zambia received a visitor from the United States. A well-to-do philanthropist, he wanted to donate a set of laptop computers to a rural school. Ignoring all advice of people on the ground, he looked around for a few days and chose a school. When it was pointed out to him that the school didn't have electricity, he just shrugged and said the laptops could be taken to the next village to charge. That that village also had a school that could have taken the laptops didn't seem relevant. Several months later he returned and was very upset to discover his laptops hadn't even been unpacked yet. The reason? None of the teachers at the school knew how to work with computers…

There were two big problems with this work of charity. First, and more obviously, the utter cluelessness of the philanthropist. Although his motivation was noble, his ignorance of the local situation and his unwillingness to listen to advice meant that a lot of money ended up being wasted. But secondly, note his assumption that a set of computers was the single best investment for a rural school in Africa. Why not invest in text books instead? Or even better, teacher education?

Education is a complex thing, but one thing pretty much all the experts agree on (along with most school children) is the supreme importance of good teachers. Yes, good teachers profit from good tools. But pupils will learn a lot more from a good teacher with bad tools than from a bad teacher with the best tools money can buy. Why is that? Because teaching is a fundamentally human enterprise. Far from being a mere transfer of knowledge, teaching is just as much about relationships and how a teacher interacts with his or her pupils. And that is something no computer can replace.

(Some will argue now that teaching is not necessary for learning to take place; and that given the right resources, students can teach themselves even in the absence of a capable teacher. To this I would reply that while that is indeed true, the percentage of school pupils able and willing to do so is negligible. And even so, they would still learn a lot more effectively if they did have a proper teacher. In short, one shouldn't base the national education strategy on this observation.)

So instead of investing it into technology (the computers), the philanthropist's money might have been better spent training teachers. This isn't easy, it isn't quick, it isn't sexy – but it works. And instead of computers that die from the dust of the African dry season or are dropped or stolen, the output of this investment is a cohort of teachers with careers spanning decades.

One project that has taken this route is FCE Masaiti, a Christian training college in Zambia. Their courses include diplomas in community development, agriculture, and teaching. Their whole approach is based around the tenet of “serving the community”. Their students live in a rural setting and are taught the skills they need for their probable future situation right from the start. When they leave, they will be government certified teachers ready to teach in any primary school in the country; and they will have been given the vision of serving their community. (Having worked alongside one of their graduates, and knowing several current students, I can attest to the quality of their training.) FCE is investing into people to help solve social problems, and their influence will be felt across the country for years to come.

* * *

“Give me a place to stand, and with a lever I will move the whole world”, said Archimedes. Having recognised the power of even a simple piece of technology to change the world, his remark is valid to this day. Technology, from the first papyrus to the printing press and the smart phone, has changed and will continue to change the way we communicate, travel, and live. What it has never once changed is human nature. It is a tremendous tool that can be leveraged for spectacular good (and just as spectacular evil). But it can never, ever, be a self-sufficient solution to a social problem.

The lever that was to move the world still needed Archimedes to operate it. As long as humans are involved – and in a social problem, they are so by definition – a technical solution will only ever be half the story. Perhaps this is most evident when working in an intercultural setting, such as development aid. Developing viable, sustainable solutions in this context requires an intimate understanding of culture: how individuals think and how communities function. Any “solutions” that pay no attention to these atechnical, human factors will never get off the ground. But it doesn't matter whether we are looking at aid work, crime reduction, or the defence of democracy; the bottom line remains the same. Societies are composed of humans, and so any workable solution to social problems – though it may also include technology – must consider the human perspective.

explodingdog: Photo



Perlsphere: Spidering websites with Headless Chrome and Selenium

Over the holidays I was working on a project that needed to download content from different websites. I needed a web spider, but the typical Perl options like WWW:Mechanize wouldn’t cut it, as with JavaScript controlling the content on many websites, I needed a JavaScript-enabled browser. But browsers consume lots of memory - what to do?

The answer was to use headless Chrome, which works exactly like normal except it has no graphical display, reducing its memory footprint. I can control it using Selenium::Remote::Driver and Selenium server. Here’s how I did it.

Non-Perl Setup

Obviously I needed to install the Chrome browser. On Linux that usually involves adding the Chrome repo, and then installing the Chrome package. On Fedora it was as easy as:

sudo dnf install fedora-workstation-repositories
sudo dnf config-manager --set-enabled google-chrome
sudo dnf install google-chrome-stable

I also needed ChromeDriver, which implements WebDriver’s wire protocol for Chrome. In other words, it’s the means by which Selenium talks with Chrome:

wget https://chromedriver.storage.googleapis.com/2.41/chromedriver_linux64.zip
unzip chromedriver_linux64.zip

I put it under /usr/bin:

sudo chown root:root chromedriver
sudo chmod 755 chromedriver
sudo mv chromedriver /usr/bin/

I downloaded Selenium server:

wget https://selenium-release.storage.googleapis.com/3.14/selenium-server-standalone-3.14.0.jar

This version of Selenium requires Java version 8, which I installed via its package:

sudo dnf install java-1.8.0-openjdk

Finally I launched Selenium server:

java -Dwebdriver.chrome.driver=/usr/bin/chromedriver -jar selenium-server-standalone-3.14.0.jar

This must be running in order for Perl communicate with Chrome using Selenium.

A basic spider

I wrote a basic spider script, here’s a simplified version:

#!/usr/bin/env perl
use Selenium::Remote::Driver;
use Encode 'encode';

my $driver = Selenium::Remote::Driver->new(
  browser_name => 'chrome',
  extra_capabilities => { chromeOptions => {args => [
    'window-size=1920,1080',
    'headless',
  ]}},
);

my %visited = ();
my $depth = 1;
my $url = 'https://example.com';

spider_site($driver, $url, $depth);

$driver->quit();

This script initializes a Selenium::Remote::Driver object. Note how it passes options to Chrome: the window-size option is an example of a key-pair option, whereas headless is a boolean. Chrome accepts a lot of options. Some others which were useful for me:

  • allow-running-insecure-content - let Chrome load websites with invalid security certificates
  • disable-infobars - disable the “Chrome is being controlled by software” notification
  • no-sandbox - disable the sandbox security feature, lets you run headless Chrome as root

The script initializes a %visited hash to store URLs the browser visits, to avoid requesting the same URL twice. The $depth variable determines how many levels deep the spider should go: with a value of 1 it will visit all links on the first page it loads, but none after that. The $url variable determines the starting web page to visit.

The spider_site function is recursive:

sub spider_site {
  my ($driver, $url, $depth) = @_;
  warn "fetching $url\n";
  $driver->get($url);
  $visited{$url}++;

  my $text = $driver->get_body;
  print encode('UTF-8', $text);

  if ($depth > 0) {
    my @links = $driver->find_elements('a', 'tag_name');
    my @urls = ();
    for my $l (@links) {
      my $link_url = eval { $l->get_attribute('href') };
      push @urls, $link_url if $link_url;
    }
    for my $u (@urls) {
      spider_site($driver, $u, $depth - 1) unless ($visited{$u});
    }
  }
}

It fetches the given $url, printing the text content of the webpage to STDOUT. It encodes the output before printing it: I found this was necessary to avoid multibyte encoding issues. If the spider hasn’t reached full depth, it gets all links on the page, and spiders each one that it hasn’t already visited. I wrapped the get_attribute method call in eval because it can fail if the link disappears from the website after it was found.

An improved spider

The spider script shown above is functional but rudimentary. I wrote a more advanced one that has some nice features:

  • Pings Selenium server on startup and quits if it’s not responding
  • Restricts the links followed to those which match the domain of the starting URL to avoid downloading content from unrelated websites
  • Converts static variables like $depth into command line options
  • Adds a debugging mode to print out the decisions made by the spider
  • Accepts a list of URLs instead of just one at a time
  • Spiders URLs in parallel using Parallel::ForkManager
  • Prints website content as gzipped files to separate content from different starting URLs and save disk space

There are other improvements I’d like to make, but those were enough to get the job done.

Trivium: 13jan2019

explodingdog: Photo



Daniel Lemire's blog: Science and Technology links (January 12th, 2019)

  1. You can buy a 512GB memory card for $140 from Amazon.
  2. We have been told for decades to avoid saturated fats, the kind found in meat, cheese and butter. After an extensive review, Grasgruber et al. conclude:

    Our results do not support the association between cardiovascular diseases and saturated fat, which is still contained in official dietary guidelines. In the absence of any scientific evidence connecting saturated fat with cardiovascular diseases, these findings show that current dietary recommendations regarding cardiovascular diseases should be seriously reconsidered.

  3. Stoet and Geary propose a new measure of gender inequality and conclude…

    We found that low levels of human development are typically associated with disadvantages for girls and women, while medium and high levels of development are typically associated with disadvantages for boys and men. Countries with the highest levels of human development are closest to gender parity, albeit typically with a slight advantage for women.

  4. Budish argues that Bitcoin would be hacked if it became sufficiently important economically.
  5. We can use deep-learning software to reconstruct intelligible speech from the human auditory cortex.
  6. Nelson’s research indicates that there will be more calories available per capita in 2050 than today, in all income quintiles, and even in an extreme climate change scenario. However, both obesity and dietery shortages (e.g., vitamin A) will remain a challenge:

    We must shift our emphasis from food security to nutrition security.

The Sirens Sound: The Aces: Girls Who Conquer The Indie Band

What comes to your mind when you hear the word “girl band”? Probably a bunch of slim ladies who could dance and sing well? This definition doesn’t apply for the muses from The Aces. They are fun, eccentric girls who bring the refreshing breeze to pop music. No dancing involved, only a great stage presence and excellent crowd controlling.

Best Friends Turns Into A Band
It should have taken a lot of effort to build a solid band that will last for years. For The Aces, two of the main members, Cristal and Alisa were interested in music making as a third grader. They are siblings who love to sing and have high interest in guitar and drum. As a child, they often sneaked into the big brother room to borrow guitar or to the neighbor’s garage to beat the drum.

One Christmas night, both of them ask their school mate, McKenna Petty to play the bass to balance their melody. It was fun as Petty had no idea how to play the bass. Three little ladies figured things out by themselves. From that day on, they built good reputation in their hometown, a small town in Utah.

Lorde’s Effect
Things went even better after they met Katie who add funky guitar tempo. Unlike three others, Katie got some skills in music. The band wouldn’t be exist if Lorde didn’t win the Grammy Awards. These four ladies are in dilemma; should The Aces disband or continue professionally? After seeing that an 18 years old could live the dream, they decided they also do. They contact Lorde’s manager and sign contract with Red Bull Records.

While the survival and talent seekers TV shows are busy to pair up different people under one roof, there are still some bandar bola groups who build the bound from the bottoms. Say it out loud: The Aces are not puppet who get together for one year and get to make music. They, without degrading the value of their friendship, make music of their own.

things magazine: The Lost Car Dealerships of London

The history of the automobile is littered with lost names, as mergers and bankruptcies constantly narrowed the field. Borgward was a German brand, surviving until 1961, making a wide range of elegant and eccentric cars, pioneering technologies like air suspension. … Continue reading

MattCha's Blog: 2018 white2tea Smoove Cocoa and Thoughts on Shortstack and Mini Tong Shu Puerh





This is busy season for me.  I’m up very early in the morning and sometimes convenience trumps careful puerh steepings this time of year.  In the wee hours of the morning, tired and still half asleep, I appreciate the ease of popping a mini puerh into my teapot to get my puerh fix…

I received a free single 7g mini puerh cake of 2018 Smoove Cocoa with my Black Friday order.  Smoove Cocoa is a shu puerh that is pressed into a coin for convenient consumption.  I really like this idea for people who are new to puerh and may be intimidated by the larger cake sizes and the often messy removal of pressed leaves from a 200g or larger size bing.  This makes more sense for a tea that is ready to consume now, like shu puerh, instead of sheng which may require aging.  You have to pay a tinny premium on price for this convenience though.  These miniscost $0.16/g for a stack of seven 7g minis vs 200g bing at $0.13/g.

Even at least 20 years ago they were pressing puerh into smaller sizes for convenience.  Dehong Tea Factory was famous for producing puerh iron pressed into 10g coins rapped in bamboo leaf and other more unconventional shapes - they have been doing it for a long time now. ( I wonder how long it will take Paul to press a melon?) In the mid-2000s I was gifted a few bamboo leaf wrapped section of these coins from the 90s/ early 2000s from Dehong Tea Factory, and it’s tasty enough, for sure.  I still have a bunch.  The very tight compression is quite deliberate as well resulting in a slow unraveling of flavours and preserving the high notes in the leaves.  They would typically do this for shu puerh.  This is just a re-imagining of this same concept in a flapjack or mini tong format.

The dry leaf smells more faint wet pile than cocoa but I can still imagine it.  It has an easy feel to it- smooth almost grainy sweet taste before turning to a very mild cocoa and wheat taste.  Almost a raisin/ currents initial taste more than cocoa.  Mild cognac taste.  Slight tight mouthfeel with moderate cooling.  Slight throat dry pulling astringent sensation which I’m never a big fan of.  Overnight infusion gives off some nice wood taste and almost berry suggestions.

Interesting that this tea named Smoove Cocoa is neither obviously chocolate tasting nor is it overly smooth for a shu puerh.  It seems that many white2tea ripe puerh have names that precondition the drinker to find a certain tastes in them but that this naming convention is not used for white2tea’s raw productions.  Is it a certain flavor that they want to curate?

On the plus, this shu has no wet pile taste thoughout and feels nice and clean in the body.  There is most definitely better and cheaper shu puerh out there, but maybe none this convenient in minis form.  I think it’s not fair that my last shu session was the famous 2017 Yunnan Sourcing Rooster King which was a significantly better shu .  I look forward to sampling other white2tea shu puerh in the future to see how their ripe quality is overall.  On the whole there is probably better shu out there for this price, I think. 


On this early cold winter morning I enjoy its warming energy just the same and appreciate the complimentary gesture.  Thanks for keepin' my morning Smmmoooove...

Peace

Disquiet: Disquiet Junto Project 0367: Trio Initiate

Each Thursday in the Disquiet Junto group, a new compositional challenge is set before the group’s members, who then have just over four days to upload a track in response to the assignment. Membership in the Junto is open: just join and participate. (A SoundCloud account is helpful but not required.) There’s no pressure to do every project. It’s weekly so that you know it’s there, every Thursday through Monday, when you have the time.

Deadline: This project’s deadline is Monday, January 14, 2019, at 11:59pm (that is, just before midnight) wherever you are on. It was posted in the late morning, California time, on Thursday, January 10, 2019.

Tracks will be added to the playlist for the duration of the project.

These are the instructions that went out to the group’s email list (at tinyletter.com/disquiet-junto):

Disquiet Junto Project 0367: Trio Initiate
The Assignment: Record the first third of an eventual trio.

Step 1: This week’s Junto will be the first in a sequence that explores and encourages collaboration. You will be recording something with the understanding that it will remain unfinished for the time being.

Step 2: The plan is for you to record a short and original piece of music, on any instrumentation of your choice. Conceive it as something that leaves room for something else — other instruments, other people — to join in.

Step 3: Record a short piece of music, roughly two to three minutes in length, as described in Step 2. When done, if possible, pan the audio so that your piece is solely in the left side of the audio.

Step 4: Also be sure, when complete, to make the track downloadable, because it will be used by someone else in a subsequent Junto project.

Seven More Important Steps When Your Track Is Done:

Step 1: Include “disquiet0367” (no spaces or quotation marks) in the name of your track.

Step 2: If your audio-hosting platform allows for tags, be sure to also include the project tag “disquiet0367” (no spaces or quotation marks). If you’re posting on SoundCloud in particular, this is essential to subsequent location of tracks for the creation a project playlist.

Step 3: Upload your track. It is helpful but not essential that you use SoundCloud to host your track.

Step 4: Post your track in the following discussion thread at llllllll.co:

https://llllllll.co/t/disquiet-junto-project-0367-trio-initiate/

Step 5: Annotate your track with a brief explanation of your approach and process.

Step 6: If posting on social media, please consider using the hashtag #disquietjunto so fellow participants are more likely to locate your communication.

Step 7: Then listen to and comment on tracks uploaded by your fellow Disquiet Junto participants.

Additional Details:

Deadline: This project’s deadline is Monday, January 14, 2019, at 11:59pm (that is, just before midnight) wherever you are on. It was posted in the late morning, California time, on Thursday, January 10, 2019.

Length: Keep the track to between two and three minutes, preferably.

Title/Tag: When posting your track, please include “disquiet0367” in the title of the track, and where applicable (on SoundCloud, for example) as a tag.

Upload: When participating in this project, post one finished track with the project tag, and be sure to include a description of your process in planning, composing, and recording it. This description is an essential element of the communicative process inherent in the Disquiet Junto. Photos, video, and lists of equipment are always appreciated.

Download: Please for this project be sure to set your track as downloadable and allowing for attributed remixing (i.e., a Creative Commons license permitting non-commercial sharing with attribution, allowing for derivatives).

For context, when posting the track online, please be sure to include this following information:

More on this 367th weekly Disquiet Junto project (Trio Initiate / The Assignment: Record the first third of an eventual trio) at:

https://disquiet.com/0367/

More on the Disquiet Junto at:

https://disquiet.com/junto/

Subscribe to project announcements here:

http://tinyletter.com/disquiet-junto/

Project discussion takes place on llllllll.co:

https://llllllll.co/t/disquiet-junto-project-0367-trio-initiate/

There’s also on a Junto Slack. Send your email address to twitter.com/disquiet for Slack inclusion.

Image associated with this project is adapted from a photo by Anna J and is used via Flickr thanks to a Creative Commons license:

https://flic.kr/p/gcojn

https://creativecommons.org/licenses/by-nc-sa/2.0/

Tea Masters: Spring and winter in TianChi

We've explored one of Taiwan's most exclusive, most beautiful and highest tea plantation: the TianChi gardens which are part of the FuShou Shan farm. Let's revisit them twice in the comfort of our Chaxi.

Let's start with the spring version harvested on May 19th.

Since I want this tea session to be educational, I have (exceptionally) weighed my dry leaves: 3 grams only. But instead of using a standard competition set, I choose to brew in a (neutral) porcelain gaiwan. I preheat it well, but I don't time my brew. I just make sure that the leaves open up well after the first brew.
Direct pour in the cups: it requires practice and skill. That's also why it's called gongfu cha!
Cristal clear brew, amazing jade color, fragrances of spring flowers and a sweet energetic taste. This is High Mountain Oolong perfection.
Spring is light, sunny and refined.

Let's turn our attention to the winter harvest. This is going to be very interesting, because these leaves come from the very same tea garden as the spring leaves above. They were harvested on October 26th, 2018. Hey! Why is a tea from end of October labeled 'winter'? For 2 reasons:
1. the Chinese lunar calendar has a different date for the seasons as the western calendar. Each season starts earlier for the Chinese. For instance, spring starts with the Chinese New Year (early February this year).
2. the growth period for tea in very high elevations is very short. The first leaves of the year are called spring and the last are called winter.
3 grams is just the right amount to cover the bottom of the gaiwan. With such an Oolong, you don't need more to enjoy its unique aromas.
The color of the open leaves is already very different. They have a yellow hue instead of a dark green one.
The color of the brew is also a little bit more yellow, but the concentration level depends also on the length of the brew.
In terms of aromas and taste, the differences with spring are very subtle. Both seasons are outstanding and share the energy of this high altitude above 2200 meters.
This winter brew is a little bit less fragrant, but has a deeper aftertaste. Maybe you'll find more nuances when brewing these 2 Oolongs.
The most obvious difference is in the spent leaves. The brews, however, can be very similar if you pay attention to the brewing. Even the last and longest brew remains clear and light.
Spring vs winter
An outstanding tea garden for spring and winter!

things magazine: Photography, above and below

Scenes from Underground, a gallery collection from The Atlantic’s In Focus feature / Metropole, a monograph about London by Lewis Bush / a collection of recent London aerial photography / maps and visualisations from Scott Reinhard / Fake Britain: A … Continue reading

Disquiet: Steady State

The machine does most of the work. It chugs along, lights blinking a telegraph of the underlying rhythm, knobs erect and at precise angles, tones rendered as held bits atmosphere, fraying as they go, the full effect a sort of aged glisten. Occasionally a hand comes into view, introducing a new note, altering the way a present sound is filtered, making other adjustments that may not be immediately evident to the listener — perhaps just to retain the work’s status quo. Sometimes when you’re the caretaker of a modular synthesizer, your job is not so much to play an instrument as it is to keep steady something that’s already moving on track, on target, in key. This video is Alex Roldan at play with his modular synthesizer, and it dates from late November of last year (earlier videos from him include drum covers of songs by Boards of Canada and Aphex Twin). Since then there have been another four modular ones from Roldan. Subscribe to his channel to encourage further endeavors.

This is the latest video I’ve added to my YouTube playlist of recommended live performances of ambient music. Video originally posted to Rodan’s YouTube channel. More from Rodan, who is based in Washington, D.C., at iamanalog.bandcamp.com.

The Shape of Code: Wanted: 99 effort estimation datasets

Every now and again, I stumble upon a really interesting dataset. Previously, when this happened I wrote an extensive blog post; but the SiP dataset was just too big and too detailed, it called out for a more expansive treatment.

How big is the SiP effort estimation dataset? It contains 10,100 unique task estimates, from ten years of commercial development using Agile. That’s around two orders of magnitude larger than other, current, public effort datasets.

How detailed is the SiP effort estimation dataset? It contains the (anonymized) identity of the 22 developers making the estimates, for one of 20 project codes, dates, plus various associated items. Other effort estimation datasets usually just contain values for estimated effort and actual effort.

Data analysis is a conversation between the person doing the analysis and the person(s) with knowledge of the application domain from which the data came. The aim is to discover information that is of practical use to the people working in the application domain.

I suggested to Stephen Cullum (the person I got talking to at a workshop, a director of Software in Partnership Ltd, and supplier of data) that we write a paper having the form of a conversation about the data; he bravely agreed.

The result is now available: A conversation around the analysis of the SiP effort estimation dataset.

What next?

I’m looking forward to seeing what other people do with the SiP dataset. There are surely other patterns waiting to be discovered, and what about building a simulation model based on the charcteristics of this data?

Turning software engineering into an evidence-based disciple requires a lot more data; I plan to go looking for more large datasets.

Software engineering researchers are a remarkable unambitious bunch of people. The SiP dataset should be viewed as the first of 100 such datasets. With 100 datasets we can start to draw general, believable conclusions about the processes involved in software effort estimation.

Readers, today is the day you start asking managers to make any software engineering data they have publicly available. Yes, it can be anonymized (I am willing to do that for people who are looking to release data). Yes, ‘old’ data is useful (data from the 1980s could have an interesting story to tell; SiP runs from 2004-2014). Yes, I will analyze any interesting data that is made public for free.

Ask, and you shall receive.

Planet Lisp: Paul Khuong: Preemption Is GC for Memory Reordering

I previously noted how preemption makes lock-free programming harder in userspace than in the kernel. I now believe that preemption ought to be treated as a sunk cost, like garbage collection: we’re already paying for it, so we might as well use it. Interrupt processing (returning from an interrupt handler, actually) is fully serialising on x86, and on other platforms, no doubt: any userspace instruction either fully executes before the interrupt, or is (re-)executed from scratch some time after the return back to userspace. That’s something we can abuse to guarantee ordering between memory accesses, without explicit barriers.

This abuse of interrupts is complementary to Bounded TSO. Bounded TSO measures the hardware limit on the number of store instructions that may concurrently be in-flight (and combines that with the knowledge that instructions are retired in order) to guarantee liveness without explicit barriers, with no overhead, and usually marginal latency. However, without worst-case execution time information, it’s hard to map instruction counts to real time. Tracking interrupts lets us determine when enough real time has elapsed that earlier writes have definitely retired, albeit after a more conservative delay than Bounded TSO’s typical case.

I reached this position after working on two lock-free synchronisation primitives—event counts, and asymmetric flag flips as used in hazard pointers and epoch reclamation—that are similar in that a slow path waits for a sign of life from a fast path, but differ in the way they handle “stuck” fast paths. I’ll cover the event count and flag flip implementations that I came to on Linux/x86[-64], which both rely on interrupts for ordering. Hopefully that will convince you too that preemption is a useful source of pre-paid barriers for lock-free code in userspace.

I’m writing this for readers who are already familiar with lock-free programming, safe memory reclamation techniques in particular, and have some experience reasoning with formal memory models. For more references, Samy’s overview in the ACM Queue is a good resource. I already committed the code for event counts in Concurrency Kit, and for interrupt-based reverse barriers in my barrierd project.

Event counts with x86-TSO and futexes

An event count is essentially a version counter that lets threads wait until the current version differs from an arbitrary prior version. A trivial “wait” implementation could spin on the version counter. However, the value of event counts is that they let lock-free code integrate with OS-level blocking: waiters can grab the event count’s current version v0, do what they want with the versioned data, and wait for new data by sleeping rather than burning cycles until the event count’s version differs from v0. The event count is a common synchronisation primitive that is often reinvented and goes by many names (e.g., blockpoints); what matters is that writers can update the version counter, and waiters can read the version, run arbitrary code, then efficiently wait while the version counter is still equal to that previous version.

The explicit version counter solves the lost wake-up issue associated with misused condition variables, as in the pseudocode below.

bad condition waiter:

while True:
    atomically read data
    if need to wait:
        WaitOnConditionVariable(cv)
    else:
        break

In order to work correctly, condition variables require waiters to acquire a mutex that protects both data and the condition variable, before checking that the wait condition still holds and then waiting on the condition variable.

good condition waiter:

while True:
    with(mutex):
        read data
        if need to wait:
            WaitOnConditionVariable(cv, mutex)
        else:
            break

Waiters must prevent writers from making changes to the data, otherwise the data change (and associated condition variable wake-up) could occur between checking the wait condition, and starting to wait on the condition variable. The waiter would then have missed a wake-up and could end up sleeping forever, waiting for something that has already happened.

good condition waker:

with(mutex):
    update data
    SignalConditionVariable(cv)

The six diagrams below show the possible interleavings between the signaler (writer) making changes to the data and waking waiters, and a waiter observing the data and entering the queue to wait for changes. The two left-most diagrams don’t interleave anything; these are the only scenarios allowed by correct locking. The remaining four actually interleave the waiter and signaler, and show that, while three are accidentally correct (lucky), there is one case, WSSW, where the waiter misses its wake-up.

If any waiter can prevent writers from making progress, we don’t have a lock-free protocol. Event counts let waiters detect when they would have been woken up (the event count’s version counter has changed), and thus patch up this window where waiters can miss wake-ups for data changes they have yet to observe. Crucially, waiters detect lost wake-ups, rather than preventing them by locking writers out. Event counts thus preserve lock-freedom (and even wait-freedom!).

We could, for example, use an event count in a lock-free ring buffer: rather than making consumers spin on the write pointer, the write pointer could be encoded in an event count, and consumers would then efficiently block on that, without burning CPU cycles to wait for new messages.

The challenging part about implementing event counts isn’t making sure to wake up sleepers, but to only do so when there are sleepers to wake. For some use cases, we don’t need to do any active wake-up, because exponential backoff is good enough: if version updates signal the arrival of a response in a request/response communication pattern, exponential backoff, e.g., with a 1.1x backoff factor, could bound the increase in response latency caused by the blind sleep during backoff, e.g., to 10%.

Unfortunately, that’s not always applicable. In general, we can’t assume that signals corresponds to responses for prior requests, and we must support the case where progress is usually fast enough that waiters only spin for a short while before grabbing more work. The latter expectation means we can’t “just” unconditionally execute a syscall to wake up sleepers whenever we increment the version counter: that would be too slow. This problem isn’t new, and has a solution similar to the one deployed in adaptive spin locks.

The solution pattern for adaptive locks relies on tight integration with an OS primitive, e.g., futexes. The control word, the machine word on which waiters spin, encodes its usual data (in our case, a version counter), as well as a new flag to denote that there are sleepers waiting to be woken up with an OS syscall. Every write to the control word uses atomic read-modify-write instructions, and before sleeping, waiters ensure the “sleepers are present” flag is set, then make a syscall to sleep only if the control word is still what they expect, with the sleepers flag set.

OpenBSD’s compatibility shim for Linux’s futexes is about as simple an implementation of the futex calls as it gets. The OS code for futex wake and wait is identical to what userspace would do with mutexes and condition variables (waitqueues). Waiters lock out wakers for the futex word or a coarser superset, check that the futex word’s value is as expected, and enters the futex’s waitqueue. Wakers acquire the futex word for writes, and wake up the waitqueue. The difference is that all of this happens in the kernel, which, unlike userspace, can force the scheduler to be helpful. Futex code can run in the kernel because, unlike arbitrary mutex/condition variable pairs, the protected data is always a single machine integer, and the wait condition an equality test. This setup is simple enough to fully implement in the kernel, yet general enough to be useful.

OS-assisted conditional blocking is straightforward enough to adapt to event counts. The control word is the event count’s version counter, with one bit stolen for the “sleepers are present” flag (sleepers flag).

Incrementing the version counter can use a regular atomic increment; we only need to make sure we can tell whether the sleepers flag might have been set before the increment. If the sleepers flag was set, we clear it (with an atomic bit reset), and wake up any OS thread blocked on the control word.

increment event count:

old <- fetch_and_add(event_count.counter, 2)  # flag is in the low bit
if (old & 1):
    atomic_and(event_count.counter, -2)
    signal waiters on event_count.counter

Waiters can spin for a while, waiting for the version counter to change. At some point, a waiter determines that it’s time to stop wasting CPU time. The waiter then sets the sleepers flag with a compare-and-swap: the CAS (compare-and-swap) can only fail because the counter’s value has changed or because the flag is already set. In the former failure case, it’s finally time to stop waiting. In the latter failure care, or if the CAS succeeded, the flag is now set. The waiter can then make a syscall to block on the control word, but only if the control word still has the sleepers flag set and contains the same expected (old) version counter.

wait until event count differs from prev:

repeat k times:
    if (event_count.counter / 2) != prev:  # flag is in low bit.
        return
compare_and_swap(event_count.counter, prev * 2, prev * 2 + 1)
if cas_failed and cas_old_value != (prev * 2 + 1):
    return
repeat k times:
    if (event_count.counter / 2) != prev:
        return
sleep_if(event_count.center == prev * 2 + 1)

This scheme works, and offers decent performance. In fact, it’s good enough for Facebook’s Folly.
I certainly don’t see how we can improve on that if there are concurrent writers (incrementing threads).

However, if we go back to the ring buffer example, there is often only one writer per ring. Enqueueing an item in a single-producer ring buffer incurs no atomic, only a release store: the write pointer increment only has to be visible after the data write, which is always the case under the TSO memory model (including x86). Replacing the write pointer in a single-producer ring buffer with an event count where each increment incurs an atomic operation is far from a no-brainer. Can we do better, when there is only one incrementer?

On x86 (or any of the zero other architectures with non-atomic read-modify-write instructions and TSO), we can... but we must accept some weirdness.

The operation that must really be fast is incrementing the event counter, especially when the sleepers flag is not set. Setting the sleepers flag on the other hand, may be slower and use atomic instructions, since it only happens when the executing thread is waiting for fresh data.

I suggest that we perform the former, the increment on the fast path, with a non-atomic read-modify-write instruction, either inc mem or xadd mem, reg. If the sleepers flag is in the sign bit, we can detect it (modulo a false positive on wrap-around) in the condition codes computed by inc; otherwise, we must use xadd (fetch-and-add) and look at the flag bit in the fetched value.

The usual ordering-based arguments are no help in this kind of asymmetric synchronisation pattern. Instead, we must go directly to the x86-TSO memory model. All atomic (LOCK prefixed) instructions conceptually flush the executing core’s store buffer, grab an exclusive lock on memory, and perform the read-modify-write operation with that lock held. Thus, manipulating the sleepers flag can’t lose updates that are already visible in memory, or on their way from the store buffer. The RMW increment will also always see the latest version update (either in global memory, or in the only incrementer’s store buffer), so won’t lose version updates either. Finally, scheduling and thread migration must always guarantee that the incrementer thread sees its own writes, so that won’t lose version updates.

increment event count without atomics in the common case:

old <- non_atomic_fetch_and_add(event_count.counter, 2)
if (old & 1):
    atomic_and(event_count.counter, -2)
    signal waiters on event_count.counter

The only thing that might be silently overwritten is the sleepers flag: a waiter might set that flag in memory just after the increment’s load from memory, or while the increment reads a value with the flag unset from the local store buffer. The question is then how long waiters must spin before either observing an increment, or knowing that the flag flip will be observed by the next increment. That question can’t be answered with the memory model, and worst-case execution time bounds are a joke on contemporary x86.

I found an answer by remembering that IRET, the instruction used to return from interrupt handlers, is a full barrier.1 We also know that interrupts happen at frequent and regular intervals, if only for the preemption timer (every 4-10ms on stock Linux/x86oid).

Regardless of the bound on store visibility, a waiter can flip the sleepers-are-present flag, spin on the control word for a while, and then start sleeping for short amounts of time (e.g., a millisecond or two at first, then 10 ms, etc.): the spin time is long enough in the vast majority of cases, but could still, very rarely, be too short.

At some point, we’d like to know for sure that, since we have yet to observe a silent overwrite of the sleepers flag or any activity on the counter, the flag will always be observed and it is now safe to sleep forever. Again, I don’t think x86 offers any strict bound on this sort of thing. However, one second seems reasonable. Even if a core could stall for that long, interrupts fire on every core several times a second, and returning from interrupt handlers acts as a full barrier. No write can remain in the store buffer across interrupts, interrupts that occur at least once per second. It seems safe to assume that, once no activity has been observed on the event count for one second, the sleepers flag will be visible to the next increment.

That assumption is only safe if interrupts do fire at regular intervals. Some latency sensitive systems dedicate cores to specific userspace threads, and move all interrupt processing and preemption away from those cores. A correctly isolated core running Linux in tickless mode, with a single runnable process, might not process interrupts frequently enough. However, this kind of configuration does not happen by accident. I expect that even a half-second stall in such a system would be treated as a system error, and hopefully trigger a watchdog. When we can’t count on interrupts to get us barriers for free, we can instead rely on practical performance requirements to enforce a hard bound on execution time.

Either way, waiters set the sleepers flag, but can’t rely on it being observed until, very conservatively, one second later. Until that time has passed, waiters spin on the control word, then block for short, but growing, amounts of time. Finally, if the control word (event count version and sleepers flag) has not changed in one second, we assume the incrementer has no write in flight, and will observe the sleepers flag; it is safe to block on the control word forever.

wait until event count differs from prev:

repeat k times:
    if (event_count.counter / 2) != prev:
        return
compare_and_swap(event_count.counter, 2 * prev, 2 * prev + 1)
if cas_failed and cas_old_value != 2 * prev + 1:
    return
repeat k times:
    if event_count.counter != 2 * prev + 1:
        return
repeat for 1 second:
    sleep_if_until(event_count.center == 2 * prev + 1,
                   $exponential_backoff)
    if event_count.counter != 2 * prev + 1:
        return
sleep_if(event_count.center == prev * 2 + 1)

That’s the solution I implemented in this pull request for SPMC and MPMC event counts in concurrency kit. The MP (multiple producer) implementation is the regular adaptive logic, and matches Folly’s strategy. It needs about 30 cycles for an uncontended increment with no waiter, and waking up sleepers adds another 700 cycles on my E5-46xx (Linux 4.16). The single producer implementation is identical for the slow path, but only takes ~8 cycles per increment with no waiter, and, eschewing atomic instruction, does not flush the pipeline (i.e., the out-of-order execution engine is free to maximise throughput). The additional overhead for an increment without waiter, compared to a regular ring buffer pointer update, is 3-4 cycles for a single predictable conditional branch or fused test and branch, and the RMW’s load instead of a regular add/store. That’s closer to zero overhead, which makes it much easier for coders to offer OS-assisted blocking in their lock-free algorithms, without agonising over the penalty when no one needs to block.

Asymmetric flag flip with interrupts on Linux

Hazard pointers and epoch reclamation. Two different memory reclamation technique, in which the fundamental complexity stems from nearly identical synchronisation requirements: rarely, a cold code path (which is allowed to be very slow) writes to memory, and must know when another, much hotter, code path is guaranteed to observe the slow path’s last write.

For hazard pointers, the cold code path waits until, having overwritten an object’s last persistent reference in memory, it is safe to destroy the pointee. The hot path is the reader:

1. read pointer value *(T **)x.
2. write pointer value to hazard pointer table
3. check that pointer value *(T **)x has not changed

Similarly, for epoch reclamation, a read-side section will grab the current epoch value, mark itself as reading in that epoch, then confirm that the epoch hasn’t become stale.

1. $epoch <- current epoch
2. publish self as entering a read-side section under $epoch
3. check that $epoch is still current, otherwise retry

Under a sequentially consistent (SC) memory model, the two sequences are valid with regular (atomic) loads and stores. The slow path can always make its write, then scan every other thread’s single-writer data to see if any thread has published something that proves it executed step 2 before the slow path’s store (i.e., by publishing the old pointer or epoch value).

The diagrams below show all possible interleavings. In all cases, once there is no evidence that a thread has failed to observe the slow path’s new write, we can correctly assume that all threads will observe the write. I simplified the diagrams by not interleaving the first read in step 1: its role is to provide a guess for the value that will be re-read in step 3, so, at least with respect to correctness, that initial read might as well be generating random values. I also kept the second “scan” step in the slow path abstract. In practice, it’s a non-snapshot read of all the epoch or hazard pointer tables for threads that execute the fast path: the slow path can assume an epoch or pointer will not be resurrected once the epoch or pointer is absent from the scan.

No one implements SC in hardware. X86 and SPARC offer the strongest practical memory model, Total Store Ordering, and that’s still not enough to correctly execute the read-side critical sections above without special annotations. Under TSO, reads (e.g., step 3) are allowed to execute before writes (e.g., step 2). X86-TSO models that as a buffer in which stores may be delayed, and that’s what the scenarios below show, with steps 2 and 3 of the fast path reversed (the slow path can always be instrumented to recover sequential order, it’s meant to be slow). The TSO interleavings only differ from the SC ones when the fast path’s steps 2 and 3 are separated by something on slow path’s: when the two steps are adjacent, their order relative to the slow path’s steps is unaffected by TSO’s delayed stores. TSO is so strong that we only have to fix one case, FSSF, where the slow path executes in the middle of the fast path, with the reversal of store and load order allowed by TSO.

Simple implementations plug this hole with a store-load barrier between the second and third steps, or implement the store with an atomic read-modify-write instruction that doubles as a barrier. Both modifications are safe and recover SC semantics, but incur a non-negligible overhead (the barrier forces the out of order execution engine to flush before accepting more work) which is only necessary a minority of the time.

The pattern here is similar to the event count, where the slow path signals the fast path that the latter should do something different. However, where the slow path for event counts wants to wait forever if the fast path never makes progress, hazard pointer and epoch reclamation must detect that case and ignore sleeping threads (that are not in the middle of a read-side SMR critical section).

In this kind of asymmetric synchronisation pattern, we wish to move as much of the overhead to the slow (cold) path. Linux 4.3 gained the membarrier syscall for exactly this use case. The slow path can execute its write(s) before making a membarrier syscall. Once the syscall returns, any fast path write that has yet to be visible (hasn’t retired yet), along with every subsequent instruction in program order, started in a state where the slow path’s writes were visible. As the next diagram shows, this global barrier lets us rule out the one anomalous execution possible under TSO, without adding any special barrier to the fast path.

The problem with membarrier is that it comes in two flavours: slow, or not scalable. The initial, unexpedited, version waits for kernel RCU to run its callback, which, on my machine, takes anywhere between 25 and 50 milliseconds. The reason it’s so slow is that the condition for an RCU grace period to elapse are more demanding than a global barrier, and may even require multiple such barriers. For example, if we used the same scheme to nest epoch reclamation ten deep, the outermost reclaimer would be 1024 times slower than the innermost one. In reaction to this slowness, potential users of membarrier went back to triggering IPIs, e.g., by mprotecting a dummy page. mprotect isn’t guaranteed to act as a barrier, and does not do so on AArch64, so Linux 4.16 added an “expedited” mode to membarrier. In that expedited mode, each membarrier syscall sends an IPI to every other core... when I look at machines with hundreds of cores, \(n - 1\) IPI per core, a couple times per second on every \(n\) core, start to sound like a bad idea.

Let’s go back to the observation we made for event count: any interrupt acts as a barrier for us, in that any instruction that retires after the interrupt must observe writes made before the interrupt. Once the hazard pointer slow path has overwritten a pointer, or the epoch slow path advanced the current epoch, we can simply look at the current time, and wait until an interrupt has been handled at a later time on all cores. The slow path can then scan all the fast path state for evidence that they are still using the overwritten pointer or the previous epoch: any fast path that has not published that fact before the interrupt will eventually execute the second and third steps after the interrupt, and that last step will notice the slow path’s update.

There’s a lot of information in /proc that lets us conservatively determine when a new interrupt has been handled on every core. However, it’s either too granular (/proc/stat) or extremely slow to generate (/proc/schedstat). More importantly, even with ftrace, we can’t easily ask to be woken up when something interesting happens, and are forced to poll files for updates (never mind the weirdly hard to productionalise kernel interface).

What we need is a way to read, for each core, the last time it was definitely processing an interrupt. Ideally, we could also block and let the OS wake up our waiter on changes to the oldest “last interrupt” timestamp, across all cores. On x86, that’s enough to get us the asymmetric barriers we need for hazard pointers and epoch reclamation, even if only IRET is serialising, and not interrupt handler entry. Once a core’s update to its “last interrupt” timestamp is visible, any write prior to the update, and thus any write prior to the interrupt is also globally visible: we can only observe the timestamp update from a different core than the updater, in which case TSO saves us, or after the handler has returned with a serialising IRET.

We can bundle all that logic in a short eBPF program.2 The program has a map of thread-local arrays (of 1 CLOCK_MONOTONIC timestamp each), a map of perf event queues (one per CPU), and an array of 1 “watermark” timestamp. Whenever the program runs, it gets the current time. That time will go in the thread-local array of interrupt timestamps. Before storing a new value in that array, the program first reads the previous interrupt time: if that time is less than or equal to the watermark, we should wake up userspace by enqueueing in event in perf. The enqueueing is conditional because perf has more overhead than a thread-local array, and because we want to minimise spurious wake-ups. A high signal-to-noise ratio lets userspace set up the read end of the perf queue to wake up on every event and thus minimise update latency.

We now need a single global daemon to attach the eBPF program to an arbitrary set of software tracepoints triggered by interrupts (or PMU events that trigger interrupts), to hook the perf fds to epoll, and to re-read the map of interrupt timestamps whenever epoll detects a new perf event. That’s what the rest of the code handles: setting up tracepoints, attaching the eBPF program, convincing perf to wake us up, and hooking it all up to epoll. On my fully loaded 24-core E5-46xx running Linux 4.18 with security patches, the daemon uses ~1-2% (much less on 4.16) of a core to read the map of timestamps every time it’s woken up every ~4 milliseconds. perf shows the non-JITted eBPF program itself uses ~0.1-0.2% of every core.

Amusingly enough, while eBPF offers maps that are safe for concurrent access in eBPF programs, the same maps come with no guarantee when accessed from userspace, via the syscall interface. However, the implementation uses a hand-rolled long-by-long copy loop, and, on x86-64, our data all fit in longs. I’ll hope that the kernel’s compilation flags (e.g., -ffree-standing) suffice to prevent GCC from recognising memcpy or memmove, and that we thus get atomic store and loads on x86-64. Given the quality of eBPF documentation, I’ll bet that this implementation accident is actually part of the API. Every BPF map is single writer (either per-CPU in the kernel, or single-threaded in userspace), so this should work.

Once the barrierd daemon is running, any program can mmap its data file to find out the last time we definitely know each core had interrupted userspace, without making any further syscall or incurring any IPI. We can also use regular synchronisation to let the daemon wake up threads waiting for interrupts as soon as the oldest interrupt timestamp is updated. Applications don’t even need to call clock_gettime to get the current time: the daemon also works in terms of a virtual time that it updates in the mmaped data file.

The barrierd data file also includes an array of per-CPU structs with each core’s timestamps (both from CLOCK_MONOTONIC and in virtual time). A client that knows it will only execute on a subset of CPUs, e.g., cores 2-6, can compute its own “last interrupt” timestamp by only looking at entries 2 to 6 in the array. The daemon even wakes up any futex waiter on the per-CPU values whenever they change. The convenience interface is pessimistic, and assumes that client code might run on every configured core. However, anyone can mmap the same file and implement tighter logic.

Again, there’s a snag with tickless kernels. In the default configuration already, a fully idle core might not process timer interrupts. The barrierd daemon detects when a core is falling behind, and starts looking for changes to /proc/stat. This backup path is slower and coarser grained, but always works with idle cores. More generally, the daemon might be running on a system with dedicated cores. I thought about causing interrupts by re-affining RT threads, but that seems counterproductive. Instead, I think the right approach is for users of barrierd to treat dedicated cores specially. Dedicated threads can’t (shouldn’t) be interrupted, so they can regularly increment a watchdog counter with a serialising instruction. Waiters will quickly observe a change in the counters for dedicated threads, and may use barrierd to wait for barriers on preemptively shared cores. Maybe dedicated threads should be able to hook into barrierd and check-in from time to time. That would break the isolation between users of barrierd, but threads on dedicated cores are already in a privileged position.

I quickly compared the barrier latency on an unloaded 4-way E5-46xx running Linux 4.16, with a sample size of 20000 observations per method (I had to remove one outlier at 300ms). The synchronous methods mprotect (which abuses mprotect to send IPIs by removing and restoring permissions on a dummy page), or explicit IPI via expedited membarrier, are much faster than the other (unexpedited membarrier with kernel RCU, or barrierd that counts interrupts). We can zoom in on the IPI-based methods, and see that an expedited membarrier (IPI) is usually slightly faster than mprotect; IPI via expedited membarrier hits a worst-case of 0.041 ms, versus 0.046 for mprotect.

The performance of IPI-based barriers should be roughly independent of system load. However, we did observe a slowdown for expedited membarrier (between \(68.4-73.0\%\) of the time, \(p < 10\sp{-12}\) according to a binomial test3) on the same 4-way system, when all CPUs were running CPU-intensive code at low priority. In this second experiment, we have a sample size of one million observations for each method, and the worst case for IPI via expedited membarrier was 0.076 ms (0.041 ms on an unloaded system), compared to a more stable 0.047 ms for mprotect.

Now for non-IPI methods: they should be slower than methods that trigger synchronous IPIs, but hopefully have lower overhead and scale better, while offering usable latencies.

On an unloaded system, the interrupts that drive barrierd are less frequent, sometimes outright absent, so unexpedited membarrier achieves faster response times. We can even observe barrierd’s fallback logic, which scans /proc/stat for evidence of idle CPUs after 10 ms of inaction: that’s the spike at 20ms. The values for vtime show the additional slowdown we can expect if we wait on barrierd’s virtual time, rather than directly reading CLOCK_MONOTONIC. Overall, the worst case latencies for barrierd (53.7 ms) and membarrier (39.9 ms) aren’t that different, but I should add another fallback mechanism based on membarrier to improve barrierd’s performance on lightly loaded machines.

When the same 4-way, 24-core, system is under load, interrupts are fired much more frequently and reliably, so barrierd shines, but everything has a longer tail, simply because of preemption of the benchmark process. Out of the one million observations we have for each of unexpedited membarrier, barrierd, and barrierd with virtual time on this loaded system, I eliminated 54 values over 100 ms (18 for membarrier, 29 for barrierd, and 7 for virtual time). The rest is shown below. barrierd is consistently much faster than membarrier, with a geometric mean speedup of 23.8x. In fact, not only can we expect barrierd to finish before an unexpedited membarrier \(99.99\%\) of the time (\(p<10\sp{-12}\) according to a binomial test), but we can even expect barrierd to be 10 times as fast \(98.3-98.5\%\) of the time (\(p<10\sp{-12}\)). The gap is so wide that even the opportunistic virtual-time approach is faster than membarrier (geometric mean of 5.6x), but this time with a mere three 9s (as fast as membarrier \(99.91-99.96\%\) of the time, \(p<10\sp{-12}\)).

With barrierd, we get implicit barriers with worse overhead than unexpedited membarrier (which is essentially free since it piggybacks on kernel RCU, another sunk cost), but 1/10th the latency (0-4 ms instead of 25-50 ms). In addition, interrupt tracking is per-CPU, not per-thread, so it only has to happen in a global single-threaded daemon; the rest of userspace can obtain the information it needs without causing additional system overhead. More importantly, threads don’t have to block if they use barrierd to wait for a system-wide barrier. That’s useful when, e.g., a thread pool worker is waiting for a reverse barrier before sleeping on a futex. When that worker blocks in membarrier for 25ms or 50ms, there’s a potential hiccup where a work unit could sit in the worker’s queue for that amount of time before it gets processed. With barrierd (or the event count described earlier), the worker can spin and wait for work units to show up until enough time has passed to sleep on the futex.

While I believe that information about interrupt times should be made available without tracepoint hacks, I don’t know if a syscall like membarrier is really preferable to a shared daemon like barrierd. The one salient downside is that barrierd slows down when some CPUs are idle; that’s something we can fix by including a membarrier fallback, or by sacrificing power consumption and forcing kernel ticks, even for idle cores.

Preemption can be an asset

When we write lock-free code in userspace, we always have preemption in mind. In fact, the primary reason for lock-free code in userspace is to ensure consistent latency despite potentially adversarial scheduling. We spend so much effort to make our algorithms work despite interrupts and scheduling that we can fail to see how interrupts can help us. Obviously, there’s a cost to making our code preemption-safe, but preemption isn’t an option. Much like garbage collection in managed language, preemption is a feature we can’t turn off. Unlike GC, it’s not obvious how to make use of preemption in lock-free code, but this post shows it’s not impossible.

We can use preemption to get asymmetric barriers, nearly for free, with a daemon like barrierd. I see a duality between preemption-driven barriers and techniques like Bounded TSO: the former are relatively slow, but offer hard bounds, while the latter guarantee liveness, usually with negligible latency, but without any time bound.

I used preemption to make single-writer event counts faster (comparable to a regular non-atomic counter), and to provide a lower-latency alternative to membarrier’s asymmetric barrier. In a similar vein, SPeCK uses time bounds to ensure scalability, at the expense of a bit of latency, by enforcing periodic TLB reloads instead of relying on synchronous shootdowns. What else can we do with interrupts, timer or otherwise?

Thank you Samy, Gabe, and Hanes for discussions on an earlier draft. Thank you Ruchir for improving this final version.

P.S. event count without non-atomic RMW?

The single-producer event count specialisation relies on non-atomic read-modify-write instructions, which are hard to find outside x86. I think the flag flip pattern in epoch and hazard pointer reclamation shows that’s not the only option.

We need two control words, one for the version counter, and another for the sleepers flag. The version counter is only written by the incrementer, with regular non-atomic instructions, while the flag word is written to by multiple producers, always with atomic instructions.

The challenge is that OS blocking primitives like futex only let us conditionalise the sleep on a single word. We could try to pack a pair of 16-bit shorts in a 32-bit int, but that doesn’t give us a lot of room to avoid wrap-around. Otherwise, we can guarantee that the sleepers flag is only cleared immediately before incrementing the version counter. That suffices to let sleepers only conditionalise on the version counter... but we still need to trigger a wake-up if the sleepers flag was flipped between the last clearing and the increment.

On the increment side, the logic looks like

must_wake = false
if sleepers flag is set:
    must_wake = true
    clear sleepers flag
increment version
if must_wake or sleepers flag is set:
    wake up waiters

and, on the waiter side, we find

if version has changed
    return
set sleepers flag
sleep if version has not changed

The separate “sleepers flag” word doubles the space usage, compared to the single flag bit in the x86 single-producer version. Composite OS uses that two-word solution in blockpoints, and the advantages seem to be simplicity and additional flexibility in data layout. I don’t know that we can implement this scheme more efficiently in the single producer case, under other memory models than TSO. If this two-word solution is only useful for non-x86 TSO, that’s essentially SPARC, and I’m not sure that platform still warrants the maintenance burden.

But, we’ll see, maybe we can make the above work on AArch64 or POWER.


  1. I actually prefer another, more intuitive, explanation that isn’t backed by official documentation.The store buffer in x86-TSO doesn’t actually exist in silicon: it represents the instructions waiting to be retired in the out-of-order execution engine. Precise interrupts seem to imply that even entering the interrupt handler flushes the OOE engine’s state, and thus acts as a full barrier that flushes the conceptual store buffer.

  2. I used raw eBPF instead of the C frontend because that frontend relies on a ridiculous amount of runtime code that parses an ELF file when loading the eBPF snippet to know what eBPF maps to setup and where to backpatch their fd number. I also find there’s little advantage to the C frontend for the scale of eBPF programs (at most 4096 instructions, usually much fewer). I did use clang to generate a starting point, but it’s not that hard to tighten 30 instructions in ways that a compiler can’t without knowing what part of the program’s semantics is essential. The bpf syscall can also populate a string buffer with additional information when loading a program. That’s helpful to know that something was assembled wrong, or to understand why the verifier is rejecting your program.

  3. I computed these extreme confidence intervals with my old code to test statistical SLOs.

OUR VALUED CUSTOMERS: To her friend...

https://vanitypublishing.bigcartel.com/

CreativeApplications.Net: Alt-C – Designing for synergy between our ecosystems and network economics

Alt-C – Designing for synergy between our ecosystems and network economics
Created by Michael Sedbon, Alt-C is an installation that uses electricity produced by plants to power a single board computer mining a cryptocurrency. The project questions our relationship to ecosystems in regards to networked technologies and abstraction problematics.

Michael Geist: Celebrating High Wireless Prices: Telus-Backed Report Claims Comparing Consumer Costs for Wireless Services is “Meaningless”

Several years ago, Telus had a message for consumers discouraged by repeated studies that found Canadians pay some of the highest wireless rates in the world. In a blog post responding to an OECD study, company executive Ted Woodhead argued “Canada really should be the most expensive country for wireless service in the Organization for Economic Co-operation and Development (OECD), but we’re not. That’s a great success story we should be celebrating.” Celebrating anything less than the world’s highest wireless prices recently came to mind as Telus  tried to sow doubt in a Canadian government commissioned study that highlighted yet again the uncompetitive realities of the Canadian wireless market. The company commissioned its own report that implausibly concludes that “communications services in Canada are cheaper than the prices foreign providers would charge for the same plans.”

In fact, the Telus backed study acknowledges that the Canadian wireless plans may leave consumers paying more, yet it argues that those “absolute costs” are meaningless:

although a Canadian plan might be more expensive in absolute terms (i.e., the monthly out-of-pocket expenses incurred by a subscriber) than a plan abroad, or vice versa, observing absolute prices is meaningless as it fails to recognize that the plans are not identical. The regression methodology adjusts (normalizes) for differences in plans and thus compares identical plans, offered on identical networks, in identical countries.

The study therefore rejects comparing actual prices for similar plans (ie. the way a consumer would compare wireless costs and affordability), opting instead to estimate what foreign carriers would charge for the same plans once a selective group of additional factors are taken into account. In doing so, it tries to incorporate the Telus view that indicia such as geography, labour costs, and the weather should be factored into a “normalized” cost for wireless services.

The Telus-backed report makes two primary claims: the Canadian government commissioned study is flawed and its own conclusions provide a better pricing comparison. It cites numerous concerns with the Canadian government commissioned study, starting with the claim that it is “meaningless” because it does not feature a testable hypothesis. It maintains that the study must start from a certain position and seek to prove or disprove the hypothesis. Yet the Canadian government commissioned study is merely a fact-gathering exercise that supports evidence-based policy making, not a report designed to advocate for a particular policy.

The report also suggests the Canadian government commissioned study is flawed because it features too few countries and does not account for network differences (speeds), country differences (geography, labour costs, weather), and differences in caller pay vs. receiver pay systems. However, some of the baskets used in the government-commissioned study include unlimited talk and text rendering the caller-pay differences irrelevant. Moreover, as noted in a post yesterday, the consistent comparison of plans enables tracking of pricing differences over many years, which show that the gap in wireless affordability is growing, not shrinking. As to a broader comparison, recent data from Tefficient suggests that expanding the comparisons only makes the Canadian situation appear worse.

With respect to its own methodology, the Telus backed report says it uses the same approach as the FCC for broadband (not general wireless services). It oddly also points to the UK’s Ofcom research, which uses price comparisons much like the Canadian government commissioned report (the Telus backed report says it is “similar but not identical”). Rejecting the actual costs consumers pay for wireless services, the report proceeds to incorporate the cherry-picked factors identified above to “forecast” what foreign providers would charge for the Canadian plans. While it says this dispels claims that Canadians pay some of the highest prices in the world, it does no such thing. It merely argues that other countries would theoretically have higher prices if they featured the same self-selected marketplace conditions as Canada.

The selective choice of factors is presumably designed to lead to higher “normalized” prices for foreign carriers. Yet the study does not include many other factors that might have the opposite effect: there is nothing on restrictions on foreign investment, nothing on mandated free roaming, nothing on the shared networks between two of Canada’s largest providers, and nothing on the competitive intensity of the market. It is almost as if the Telus study picked the factors designed to “prove” its hypothesis.

While Telus undoubtedly hopes to use the study to lobby against much-needed reforms to inject competition into a sector that even the Competition Bureau says raises concerns with market power, the study is more likely to be viewed by the government and consumers as a partisan document (the references to MVNO policy is a giveaway) that does little to temper lingering frustration with high wireless pricing and the effects of limited competition. Given that Canadian consumers pay actual bills – not normalized ones subject to regression analysis – the data from the vast majority of studies that point to Canadians facing some of the highest wireless prices in the world still stands.

The post Celebrating High Wireless Prices: Telus-Backed Report Claims Comparing Consumer Costs for Wireless Services is “Meaningless” appeared first on Michael Geist.

MattCha's Blog: 2018 white2tea Snoozefest & the Real Meaning of Snoozefest


Sometimes, probably because of my age or just how I see the world or because I’ve been at this tea thing before some of you readers were born, I don’t get it.  Portions of my position and criticism of the 2017 white2tea Snoozefest is one such instance found here on this blog…

In this post in 2017, I laid out the reason why I thought the marketing around the Snoozefest puerh was a joke.  The argument I initially laid out is as follows…

There is no way of determining if this is actually a $40 cake offered at $15, because it is in fact offered at $15.  Also, there is no way of sampling it before you buy to actually test this argument and determine for yourself if it is actually worth $40 (or at least $15) because it is offered as a limited run and intended to sell out very quickly.  Nor can you compare to other puerh of $40.00 ($0.20/g) or even $15.00 ($0.08/g) in this category because you only have one shot, a gamble really, to determine if this one is truly the bargain white2tea claims or just some marketing stunt.

Although my feelings about this criticism was honest at the time and some of it still holds true (some of it can just as easily be applied to the 50 cake limited 2018 Tunji which sold out in under 18 hours), my position on this has changed over the last year because of two realities that I have come to accept in regards to puerh in 2018…


I know this is nothing new but I realized this year that it is extraordinarily rare how many fresh young puerh cakes are offered at very low prices (below $0.10/g).  And out of these how many are actually good?  All products have a low point which the price must be above to turn a profit or at which the cost of production doesn’t exceed the price at which it is offered.  I wonder in 2018 how low that point is?  It will be different for each producer, for sure, based on their business model and mark up.


This year I realized that it is a common marketing strategy these days, for one reason or another, to offer a very limited amount of puerh that is basically meant to sell out before anyone has tried it.  This is opposite the traditional model of selling puerh where a very large volume of puerh is pressed as both an investment to age in their warehouse as well as to simply have enough in stock to sell as much as they can possibly can.

Besides coming to these conclusion the third thing that I have resolved within myself is that Autumn puerh can in fact be good puerh and worthwhile drinkers.  I have opened myself up to the possibility of purchasing of Autumn puerh partly in response to increasing prices.

So after coming to terms with these realities, I realize that “Snoozefest” has a different meaning all together.  Offering a puerh cake at $0.08/g of fresh puerh has to be pushing that lower profit limit for most of our western puerh vendors like white2tea.  For Yunnan Sourcing, a more traditional western puerh vendor, they still manage to offer a handful of these every year and probably do it within their pre-existing profit margins.  Scott of Yunnan Sourcing claims that he prices all puerh on a formula not on perceived value.  So, I now feel that offering fresh puerh at $0.08/g is probably a gift in and of itself.  Something, I failed to understand a year ago.

The name “Snoozefest” refers to the flash sale offering of this cake.  It sold out fast this year (an hour or so) and almost crashed the white2tea site.  If you snooze, you loose (out).  The rapper this year has added some additional features on to last year’s wrapper (my interpretation is here link).  It also has other pictures and features that give the cake a hyped and gangsa-esque feeling.  The thief motif of the free tote that came with the purchase kind of pairs with the feeling of this cake.  The thief motif holds special importance in urban art, specifically in graffiti art.  This Banksy inspired design speaks to both the reclamation of public space and discourse (pertaining to tea) that white2tea is branded towards as well as the rebel, going against the norm or establishment, positioning of the brand.  This is a very nice touch and esthetic that I appreciate with my purchase.  In another marketing stunt, there are apparently at least 2 completely different totes that were given away.

So… I knew I was going to purchase a white2tea cake this year that I’ve had my eye on.  Also I was planning on sampling 2018 white2tea Splendid in a new quest- a search for the best cheapest young puerh (much more on this in a coming post).  When I went to the white2tea site it had grinded down to a snail’s pace and I ended up getting out with a number of these Snoozefest cakes as well as the cake I initially went in for but pulled the plug before picking up a 2018 Splendid sample.  I was concerned about my cart crashing.  I also was a bit surprised as there was little forewarning that Paul was going to drop another Snoozefest cake, were you?

My rationale for picking up a decent number of 2018 Snoozefest cakes is the following…

Reviews of the 2017 cake seem to be positive from what I read- vegetal, savory, slight bitter, astringent, sweet, creamy, floral, pine with the negative being a bit dry in the mouth and lacking stamina (see here and here and here).  Some comments suggested that people wished they had ordered more.

My other rationale is that this is the absolute cheapest white2tea sheng puerh and, as stated above, I am on a mission to find the best of the cheapest stuff.  My reasoning above about the rising cost of maocha is enough reason to purchase.

Finally, I hope this purchase will test out the claims made by whtie2tea that this tea is actually a $40.00 tea not the $15.00 they sell it for.  I have only sampled a few mid-priced white2tea sheng offerings and I really want to see what the bottom of their brand is like.  Let’s get to it…

Dry leaves smell of distant sweet fruits and florals, it’s a faint but distinctly fruity and sweet smell.  I pack a bunch into the pot.

First infusion has a muted woody almost salty barely floral/sweet ghostly candy onset there is a soft suggestion of flat sugar sweetness and some vegetal taste.  The aftertaste slowly and softly pops florals and candy floss.  The mouthfeel is silky with a slight stickiness in the mouth.

The second infusion has a soft almost sweet pea and pungent muted candy floss initial taste over a slightly salty dry wood.  It has a nice fragrant entry and soft fluffy mouthfeeling.  The cooling throat feel is mild and there is a slight rubbery sensation in the aftertaste over mutted cotton candy and plums.  This blend shows lots of different elements in it.  The mouthfeeling is light and sticky, the lips feel dry.

The third infusion has a tangy almost grapefruit and wildflower/ candy floss initial taste with a salty/ savory approach.  There is a creamy sweet, candy like, and savory woody aftertaste.  The mouthfeel has a distinct astringent feeling and makes the teeth feel squeaky.  The mild cooling and muted candy aftertaste is long and returns minutes later.

The fourth infusion has a strong initial taste of fragrant florals, sour citrus, plum, muted candy sweetness, saltiness over as slightly astringent and bitter base.  The mouthfeel is astringent and slightly sticky.  It pushes the saliva into the mid- and upper-throat.  The aftertaste is long and is held by the strong mouthfeeling and throat feeling.

The fifth infusion starts with a sour and salty grapefruit like initial taste which turns into a floral nuanced thing.  It then turns creamy, chalky and woody with a candy-like in its aftertaste.  It has a certain level of astringency and is slightly bitter.

The sixth infusion has a more pronounced bitter and astringency to it but a more upfront very salty, sour and floral profile.  The aftertaste is long and sweet the astringency pushes saliva in the throat and hold the creamy sweet, dry wood, and candy aftertaste in place.  The mouthfeel is a balance of pucker and almost stickiness.

The seventh infusion starts with a creamy sweet candy like sweetness, almost soapiness, in a dry woody and astringent base with heavier floral suggestions.  The liquor isn’t overly thick in the mouth and the taste isn’t overly deep but the plethora of high and complex notes are held in tightly by the astringency with capture them all in the throat nicely.  It’s almost as if this tea lack a mid-profile, or dense grounding thickness but is compensated by complicated interplay of highnotes, and good mouthfeeling/ throatfeeling.  The qi of this tea is mild with a fuzzy/ muffled head feeling, mild relaxing and slight alertness.  In the body you can feel a subtle heaviness in the solar plexus between the heart and stomach area.  It also has a mildly relaxing effect on the shoulders.  Got some very mild itch with this tea also on the legs, allergy reaction to this one which I have never experienced before with white2tea.

The eighth has a fruity sweet burst initially with barely salty and creamy sweet overtures.  There is a sticky sweet long lingering candy aftertaste.  The taste here is much more sweet and simple and the bitter/ astringency in this infusion is much less.

The ninth infusion starts with a slightly sour juicy fruity initial sweetness ends in a long creamy sweetness.  There is a notable grapefruit/citrus flavor in the intial taste and a creamy sweet finish.  The mouthfeel is sticky, lips drying, even slightly sandy.

The tenth infusion has a bitter onset with a grapefruit and heavier perfume floral arrangement there is still that creamy sweetness and a touch more vegetal taste as well.  The aftertaste is long and the bitterness and astringency is notable.  The mouhtfeeling is pucker.

11th starts creamy sweet, slightly sour, astringent, slightly bitter orange peel.  The aftertaste is floral and long creamy sweetness that turns into slight sour, bitter almost dry wood. Citrus peeling, cotton candy, flowershop.  Qi is slow to build but I feel it in many different places. Mild itch.  Medium young puerh harshness in Stomach.  This tea is not meant to drink now, this is surprising to me.  Last year’s reviews suggested something milder with less stamina and more drink now, from what I read.  This year’s Snoozefest is not that.

12th is bitter, slight floral faint woodiness underneath.  The aftertaste is long and creamy sweet.

The 13th is a mild, creamy floral woodiness, with sticky mouthfeeling and sweet creamy aftertaste which starts to disappear into woody vegetal tastes.

14th is more bland wood with most of the sweetness, the creamy type, in the aftertaste.

15th I add 15 seconds to the flash infusions and push out much more sweet creamy tastes initially then mouthfeel is less astringent now and more dry sticky lips and slightly sandy in the mouth with a fruit like taste lingering in the aftertaste.

The 16th is steeped for 30 seconds again and has a sour fruity initial with creamy sweet tastes and present enough mouthfeel even this late in the session.

I throw in the towel with this tea but it is enjoyable enough to continue, I just don’t have the time today so it goes into an overnight steeping.

Decent Stamina here with enjoyable flavours late into the gong fu session with mouthfeel still holding on. Nice.

The positives with this tea is that it has a nice mouthfeeling that really hold the complex blend of interesting highnotes together nicely.  It has enough diversity from steep to steep to keep it interesting.  Its stamina- this tea actually can be steeped longer than the average sheng.  The Qi is nothing special but enjoyable.  Overall it lacks a thick/deep feeling in the mouth and the tastes feels more superficial a character more common of autumn puerh.  Another negative is that it is the first white2tea sheng to trigger very mild allergy response from me but it was the mildest I have felt to date so I can probably overlook it.

Overall, I have the impression that it is more of a $40.00 cake than a $15.00 one so I am quite happy with my purchase.  My assessment is that this year's 2018 seems better than the notes I've read on the 2017.  I taste mainly Yiwu and Jinggu, and maybe even Southern Menghai.  This is mostly, if not completely, autumnal picked puerh in this blend.

Peace

Edit Jan 9, 2019: I have had a chance to drink this again a few times with harder water and in my large 200ml Yixing.  It is still a bit hard on the system being so young but yet requires a full stuffing of the pot with leaves to bring out the full brilliant high notes of this tea.  However, the more leaves you use the harsher more astringent bitter it becomes.  If you use less leaf it just becomes insipid- what's the point of that?  I recommend soft water in a very small teapot stuffed full of leaves for the best effect. 

In my search for the best cheapest young sheng puerh I have to ask the question, "Would I order more for $15.00?  Maybe try for it next year?  I keep thinking that I likely would not mainly because the qi is just too weak in this cake.  The top notes and astringency work nice together though.

I am going to wrap up this tong and put it in pretty dry storage to preserve those nice notes.  In ten years the harshness will be diminished but the top will remain... see you in 10 years...

Double Peace

OUR VALUED CUSTOMERS: While discussing aging...

https://vanitypublishing.bigcartel.com/

things magazine: Gone but not

Small snippets today. A list of people who claimed to be Jesus / you are getting old / ambient music from The Leaf Library / the world’s population depicted in 3D / the decline of

Michael Geist: More Steps Needed: Government Commissioned Report Shows Canadian Wireless Pricing Remains Among Highest in the Developed World

The Canadian government released the 2018 price comparison of wireless pricing just before the holidays, promoting the report with a press release trumpeting “greater competition leads to reduced mobile wireless price plans for Canadians.” Despite the optimism from Innovation, Science and Economic Development Minister Navdeep Bains, a closer look at the data shows that Canadians continue to pay some of the highest wireless prices in the world. In fact, a comparison of pricing changes since the Liberals won the 2015 election reveals that Canada lags badly behind peer countries in the reduction of pricing of common wireless plans.

Consider the two baskets (or levels) that many consumers encounter when they consider a wireless plan: unlimited talk and text plus either 2 GB or 5 GB of data. In each case, Canadian prices are either the highest or second highest among the reviewed countries. Not only are the prices high, they are typically falling more slowly than in those other countries. In other words, the gap between Canada and other countries on wireless affordability is growing, not shrinking.

Level 4 – 2 GB, unlimited talk and text

Country
2015
2018
Percent change
Canada 83 75 -9%
US 92 61 -34%
UK 62 27 -56%
France 48 31 -35%
Germany 72 46 -36%
Italy 63 21 -67%
Australia 78 25 -68%
Japan 82 NA

 

Level 5 – 5 GB, unlimited talk and text

Country
2015
2018
Percent change
Canada 107 87 -28%
US 131 98 -25%
UK 69 31 -55%
France 61 34 -44%
Germany 103 65 -37%
Italy 77 30 -61%
Australia 97 27 -72%
Japan 103 NA

 

While the Canadian carriers seem determined to adopt a climate change denier style approach by injecting dubious data into the debate (more on that in a post tomorrow), the reality is that comparison data overwhelmingly points to Canadian wireless prices as uncompetitive, leading to reduced usage and harming the innovation economy. As Meghan Sali points out in her excellent Globe and Mail op-ed, the lack of wireless competitiveness is emerging as a political issue. With an election only months away, the time for real policy reform – including mandated MVNOs to encourage greater competition – is long overdue.

The post More Steps Needed: Government Commissioned Report Shows Canadian Wireless Pricing Remains Among Highest in the Developed World appeared first on Michael Geist.

OCaml Weekly News: OCaml Weekly News, 08 Jan 2019

  1. Opam 2 ppa available for Ubuntu Trusty/Xenial/Bionic/Cosmic now
  2. Interesting OCaml Articles
  3. opam 2.0.2!
  4. Other OCaml News

Disquiet: A Jigsaw of Desire

If you really love a piece of music, you don’t just remember it in sequence. You remember it in slices, bits that your brain plays on repeat, often without effort. You remember it in different keys. You remember it slower, faster, in a different time signature altogether. You remember segments played out of the original order, layered, yoked together through a compulsion for a different whole, or more to the point: in a new arrangement that prioritizes your ideal comprehension of the song, a jigsaw of desire.

To hear Alice Coltrane’s “Om Shanti” re-worked (the hyphen added to emphasize the newness, the apartness-from-the-original) by Peter Speer on the sparest of a modular synthesizer setup — just two pieces: a sampler and a tool to control the sampler — is to hear such a mental remapping, albeit here performed in realtime. It’s Speer’s consciousness manifesting in the physical world of dials and cables. Coltrane’s alternately sultry and mournful tonalities are stretched and echoed, turned into nano-washboard rhythms and deep cavernous spaces, all in fitting tribute to her otherworldly oeuvre. Grounding the effort are the modest, even mundane, maneuvers that Speer must enact over the course of the video to accomplish his goals.

Video originally posted at vimeo.com. More from Speer at instagram.com/peter.speer.

Planet Lisp: Zach Beane: Converter of maps from Reflex Arena to QuakeWorld

Converter of maps from Reflex Arena to QuakeWorld:

Via dzecniv on reddit.

Disquiet: Algorithmic Art Assembly

My friend Thorsten Sideb0ard is hosting Algorithmic Art Assembly, a new event in San Francisco on March 22nd and 23rd this year, “focused on algorithmic tools and processes.” I’ll be doing a little talk on the 22nd, which is a Friday.

Speakers include: Windy Chien, Jon Leidecker (aka Wobbly), Julia Litman-Cleper, Adam Roberts (Google Magenta), Olivia Jack; Mark Fell (a Q&A), Spacefiller, Elizabeth Wilson, M Eiffler, Adam Florin, Yotam Mann & Sarah Rothberg — and me. Performances include: Kindohm, Algobabez, Renick Bell, Spatial, Digital Selves, Wobbly, Can Ince; Mark Fell, W00dy, TVO, Shatter Pattern, William Fields, Sebastian Camens, Spednar. Here’s a bit more from the website, aaassembly.org:

Algorithmic Art Assembly is a brand new two day conference and music festival, showcasing a diverse range of artists who are using algorithmic tools and processes in their works. From live coding visuals and music at algoraves, to virtual reality, gaming, augmented tooling, generative music composition, or knot tying, this event celebrates artists abusing algorithms for the aesthetics.

Daytime talks will present speakers introducing and demonstrating their art, in an informal and relaxed setting, (very much inspired by Dorkbot).

Each day will feature one workshop in an intimate setting, creating an opportunity for you to learn how to create live coded music using two of the main platforms, SuperCollider and TidalCycles. Workshops are limited in space, with reservation required – details to come.

Evening performances will be heavily based upon the algorave format, in which the dancefloor is accompanied by a look behind the veil, with several artists projecting a livestream of their code on screen. Performers will play energetic sets back to back, with minimal switch-over time.”

It was a new year, so I cleaned up my bio a bit. Here’s how it reads currently:

Marc Weidenbaum founded the website Disquiet.com in 1996 at the intersection of sound, art, and technology, and since 2012 has moderated the Disquiet Junto, an active online community of weekly music/sonic projects that explore creative constraints. A former editor of Tower Records’ music magazines, Weidenbaum is the author of the 33 1⁄3 book on Aphex Twin’s classic album Selected Ambient Works Volume II, and has written for Nature, Boing Boing, Pitchfork, Downbeat, NewMusicBox, Art Practical, The Atlantic online, and numerous other periodicals. Weidenbaum’s sonic consultancy has ranged from mobile GPS apps to coffee-shop sound design, comics editing for Red Bull Music Academy, and music supervision for two films (the documentary The Children Next Door, scored by Taylor Deupree, and the science fiction short Youth, scored by Marcus Fischer). Weidenbaum has exhibited sound art at galleries in Dubai, Los Angeles, and Manhattan, as well as at the San Jose Museum of Art, and teaches a course on the role of sound in branding at the Academy of Art University in San Francisco. Weidenbaum has commissioned and curated sound/music projects that have featured original works by Kate Carr, Marielle V Jakobsons, John Kannenberg, Steve Roden, Scanner, Roddy Schrock, Robert Thomas, and Stephen Vitiello, among many others. Raised in New York, Weidenbaum lives in San Francisco.

More on the Algorithmic Art Assembly at aaassembly.org. The event will take place, both days, at Gray Area Foundation for the Arts grayarea.org.

Planet Lisp: Quicklisp news: January 2019 Quicklisp dist update now available

New projects: 
  • cl-markless — A parser implementation for Markless — Artistic
  • data-lens — Utilities for building data transormations from composable functions, modeled on lenses and transducers — MIT
  • iso-8601-date — Miscellaneous date routines based around ISO-8601 representation. — LLGPL
  • literate-lisp — a literate programming tool to write common lisp codes in org file. — MIT
  • magicl — Matrix Algebra proGrams In Common Lisp — BSD 3-Clause (See LICENSE.txt)
  • nodgui — LTK — LLGPL
  • petri — An implementation of Petri nets — MIT
  • phoe-toolbox — A personal utility library — BSD 2-clause
  • ql-checkout — ql-checkout is library intend to checkout quicklisp maintained library with vcs. — mit
  • qtools-commons — Qtools utilities and functions — Artistic License 2.0
  • replic — A framework to build readline applications out of existing code. — MIT
  • slk-581 — Generate Australian Government SLK-581 codes. — LLGPL
  • sucle — Cube Demo Game — MIT
  • water — An ES6-compatible class definition for Parenscript — MIT
  • winhttp — FFI wrapper to WINHTTP — MIT
Updated projects3d-matrices3d-vectorsaprilasd-generatorchirpchronicitycl-asynccl-batiscl-collidercl-dbicl-dbi-connection-poolcl-enumerationcl-formscl-hamcrestcl-hash-utilcl-lascl-libevent2cl-libuvcl-mixedcl-neovimcl-openglcl-patternscl-punchcl-satcl-sat.glucosecl-sat.minisatcl-syslogcl-unificationcladclazyclimacsclipcloser-mopcroatoandbusdeedsdefenumdefinitionsdufyeasy-bindeasy-routeseclectoresrapf2clflareflexi-streamsflowgendlglsl-toolkitharmonyhelambdaphu.dwim.debughumblerinquisitorlakelegitlichat-protocollisp-binarylisp-chatlog4cllqueryltkmcclimnew-opomer-countookoverlordpetalisppjlinkplumppostmodernprotestqtoolsquery-fsratifyread-numberrpcqsafety-paramssc-extensionsserapeumslimeslyspecialization-storespinneretstaplestatic-dispatchstumpwmsxqltootertrivial-clipboardtrivial-socketsutilities.print-itemsvernacularwebsocket-driverwild-package-inferred-systemxhtmlambda.

Removed projects: cl-llvm, cl-skkserv

The removed projects no longer build for me.

To get this update, use (ql:update-dist "quicklisp"). Enjoy!

The Sirens Sound: Why You Should Love This British Indie Band

There are lots of great indie bands from the UK and one of them is London Grammar. Not many people in the world know this indie band well, but London Grammar has gained more popularity recently through their songs “Strong” and “Hey Now”. If you are a fan of indie music, you definitely need to listen to London Grammar’s songs.

Hannah Reid’s Great Vocal Range
Indie bands have their own characteristics and London Grammar also has its own color. The unique color of their music is felt through the vocal range of the vocalist, Hannah Reid. She has a deep voice and lots of music critique praise her vocal range. She has a contralto vocal range, which is very rare, and it helps in shaping the color of their music.

Great Songs with Deep Meaning
The best thing about this band is that they always create stories through their songs. The lyrics are very deep as they come from their own experience. Most of them imply sadness and they go well with the melancholic melody. As you listen to the songs, you will feel the same experience and it will soothe you in such a way.

Distinct Musical Style
The songs from London Grammar are great for those who enjoy a calm atmosphere. Some people will find it a little bit boring to listen to their songs, but as you listen more, you will be swayed with the melancholic arrangement. It is also good to listen to their songs before bedtime since you will feel calmer and you can sleep soundly.

The soothing voice of the vocalist will enable you to create a good “grammar” for your soul. Their music is always deep in meaning and it can be a good idea to listen to the songs as your goodnight lullaby.

Daniel Lemire's blog: Science and Technology links (January 5th, 2019)

  1. There are nearly 70,000 centenarians in Japan.
  2. China’s population fell by 1.27 million in 2018.
  3. Obesity is associated with increased senescent cell burden and neuropsychiatric disorders, including anxiety and depression. Clearing senescent cells could alleviate these symptons.
  4. Women who come from families where the mother is the breadwinner are less likely to pursue a science degree at university level.
  5. Stronger men live longer:

    Muscular strength is inversely and independently associated with death from all causes and cancer in men, even after adjusting for cardiorespiratory fitness and other potential confounders.

  6. Inflammation is a crude but effective part of our immune system that serves as a first line of defense. It is necessary but tends to get out of hand as we age. Arai et al. found that your level of inflammation was a potential predictor of how likely you were to reach extreme old age:

    Inflammation is the prime candidate amongst potential determinants of mortality, capability and cognition up to extreme old age

  7. An article published by Nature argues for rejuvenation strategies:

    Ageing is associated with the functional decline of all tissues and a striking increase in many diseases. Although ageing has long been considered a one-way street, strategies to delay and potentially even reverse the ageing process have recently been developed. Here, we review four emerging rejuvenation strategies—systemic factors, metabolic manipulations, senescent cell ablation and cellular reprogramming—and discuss their mechanisms of action, cellular targets, potential trade-offs and application to human ageing.

  8. Childhood leukaemia is a feature of developed societies and has to do with the fact that children are insufficiently exposed to bacteria and infections.
  9. China has landed a rover on the far side of the Moon.
  10. While climate change models predict that as the world warms, biomass will decompose more quickly, which would send a lot more carbon dioxide into the atmosphere, new research finds that it may work the other way around: warmer temperature may reduce decomposition and carbon emissions.
  11. You can defeat Google’s captchas (meant to prove you are a human being) by taking the audio test and feeding it back to Google’s services to get a transcription.

The Sirens Sound: Parquet Courts: The Raising Rock Indie Band From NY

Ever heard of “rock never dies” phrase? Well, it really does. Just take a look at the raise of new rock band from America; Parquet Courts. They bring rock music with different feels, charging without worrying about the mixed genre slipped in. It seems like there is no boundaries of their rock genre exploration.

The Background
It was 2010 when this band was formed. The original members are Andrew Savage, Fergus and Geronimo, Austin Brown, Sean Yeaton and Max Savage. In charge of vocal and guitar are Andrew Savage and Austin Brown. The drummer is Max Savage and Sean Yeaton is the bassist. The other two members, Fergus and Geronimo departs from the group before it gets to be famous.

Their first album is entitled “American Specialties”. It is doing well in the chart, but not well enough to catch the attention of the fans. It makes the band works harder to explore the style and their own color of rock music. In 2012, they release the second album “Light Up Gold” which catch the attention of many indie band magazines.

Wide Awake!
The gold year for the whole Parquet Courts career is definitely in 2018. Their album “Wide Awake” has sweet selling rates right after it was released to the market. This is without doubt the best album of the band. Everything about the songs, lyrics, composition, even the music video are in sync. Thanks to the friendly and well known director, Danger Mouse, they get extra hands to bring their music to perfection.

Six years is not a long time to gain fame, especially as a rock band. Although this genre has its own fans, not every band could gain mass as fast as Parquet Courts does. Probably one of the determining factor is the endless exploration of the music.

Better Embedded System SW: Counter Rollover Brings Down Rail Service

In October 2018 Hong Kong had "six hours of turmoil" in their rail service due to as signalling outage. The culprit has now been identified as counter roll-over.

South China Morning Post
https://www.scmp.com/news/hong-kong/transport/article/2178723/unknown-signalling-system-incompatibility-caused-october

Summary version: a system synchronization counter had been counting away since 1996 and required a system reset when it saturated.  (At least it didn't just roll over without anything noticing.)  But over the years two different systems with slightly different counter roll-over procedures were installed.  When rollover time came, they disagreed with each other on count value, paralyzing the system during the window until the second system shut down due to counter saturation.  Details below quoted from the official report. (https://www.mtr.com.hk/archive/corporate/en/press_release/PR-18-108-E.pdf)

The Detailed version:
"5.1.3. Data transmission between sector computers is always synchronized through an internal software counter in each sector computer. If any individual sector computer is individually rebooted, its counter will be re-initialized and will immediately synchronize to the higher counter figure for the whole synchronized network. Therefore, when the Siemens sector computers were commissioned and put into service in 2001/2002, the relevant counters were synchronized to those of the Alstom sector computers which were installed in 1996. If the counter reaches its ceiling figure, the associated sector computer will halt and need to be re-initialized. However the counter re-initialization arrangements for the two suppliers’ sector computers are different. The Alstom sector computers will be re-initialized automatically once their counters reach an inbuilt re-initialization triggering point approximately 5 hours before reaching the ceiling figure. However, this internal software function was not made known to the operators and maintainers. The Siemens sector computers do not have an automatic reinitialization function and therefore need to be manually reinitialized through rebooting in SER by maintenance staff.  
5.1.4 At around 05:26 hours on the incident day, the Alstom software counters reached the triggering point for automatic re- initialization while the Siemens sector computers continued counting up, creating an inconsistent re-initialization situation between the two interconnected sector computers at KWT (Alstom) and LAT (Siemens). This resulted in repeated execution of re-initialization followed by re-synchronization with the higher counter figure from LAT, in the KWT sector computer in an endless loop causing corresponding instability in all 25 Alstom sector computers in the system.  
5.1.5 When all the Siemens software counters reached the ceiling figure at around 10:22 hours, some 5 hours after the Alstom sector computers had passed their automatic re-initialization triggering point, the 8 Siemens sector computers halted as designed. Moreover, trains on the TKL had already encountered trainborne signalling failure earlier at 10:02 hours due to the around 20 minutes counter look ahead validity requirements. 
5.1.6 After the interconnections between the signalling systems of the relevant lines and the Alstom and Siemens sector computers between KWT and LAT were isolated, all sector computers were effectively rebooted to complete the entire re-initialization process and the signalling system for the four incident lines resumed normal. "
With credit for calling my attention to the report to:
Date: Sun, 30 Dec 2018 15:39:37 +0800
From: Richard Stein 
Subject: Re: MTR East Rail disruption caused by failure of both primary 
 and backup (Stein, RISKS-30.89)

The Shape of Code: Changes in the shape of code during the twenties?

At the end of 2009 I made two predictions for the next decade; Chinese and Indian developers having a major impact on the shape of code (ok, still waiting for this to happen), and scripting languages playing a significant role (got that one right, but then they were already playing a large role).

Since this blog has just entered its second decade, I will bring the next decade’s predictions forward a year.

I don’t see any new major customer ecosystems appearing. Ecosystems are the drivers of software development, and no new ecosystems has several consequences, including:

  • No major new languages: Creating a language is a vanity endeavor. Vanity project can take off if they are in the right place at the right time. New ecosystems provide opportunities for new languages to become widely used by being in at the start and growing with the ecosystem. There is another opportunity locus; it is fashionable for companies that see themselves as thought-leaders to have their own language, e.g., Google, Apple, and Mozilla. Invent your language at the right time, while working for a thought-leader company and your language could become well-known enough to take-off.

    I don’t see any major new ecosystems appearing and all the likely companies already have their own language.

    Any new language also faces the problem of not having a large collection packages.

  • Software will be more thoroughly tested: When an ecosystem is new, the incentives drive early and frequent releases (to build a customer base); software just has to be good enough. Once a product is established, companies can invest in addressing issues that customers find annoying, like faulty behavior; the incentive change results in more testing.

    There are other forces at work around testing. Companies are experiencing some very expensive faults (testing may be expensive, but not testing may be more expensive) and automatic test generation is becoming commercially usable (i.e., the cost of some kinds of testing is decreasing).

The evolution of widely used languages.

  • I think Fortran and C will have new features added, with relatively little fuss, and will quietly continue to be widely used (to the dismay of the fashionista).
  • There is a strong expectation that C++ and Java should continue to evolve:

    • I expect the ISO C++ work to implode, because there are too many people pulling in too many directions. It makes sense for the gcc and llvm teams to cooperate in taking C++ in a direction that satisfies developers’ needs, rather than the needs of bored consultants. What are Microsoft’s views? They only have their own compiler for strategic reasons (they make little if any profit selling compilers, compilers are an unnecessary drain on management time; who cares what happens to the language).
    • It is going to be interesting watching the impact of Oracle’s move to charging for runtimes. I have no idea what might happen to Java.

In terms of code volume, the future surely has to be scripting languages, and in particular Python, Javascript and PHP. Ten years from now, will there be a widely used, single language? People have been predicting, for many years, that web languages will take over the world; perhaps there will be a sudden switch and I will see that the choice is obvious.

Moore’s law is now dead, which means researchers are going to have to look for completely new techniques for building logic gates. If photonic computers happen, then ternary notation may reappear again (it was used in at least one early Russian computer); I’m not holding my breath for this to occur.

Tea Masters: Dong Ding Dong Ding Dong Ding

Have you heard the bell ring? Christmas vacation is over and we're back in tea class! Sarah and Rattana are back in Taiwan and have asked me to teach them about Dong Ding Oolong. So, I chose several teas to explain what Dong Ding tea is all about.
We started with this Hung Shui Oolong from Feng Huang, winter 2017. This is an excellent example of what a Dong Ding Oolong is and should taste like. The cultivar is Qingxin Oolong and it's grown in Feng Huang, one of the villages that are part of the rather small Dong Ding area. The winter season and the spring season both see the Dong Ding Oolong take place. It was created in 1976 by the director of the Taiwan Research and Extension Station (TRES, Taiwan's tea research institute) to raise quality by using a standard taste close to that of WuYi Oolongs (which were not available in Taiwan at the time, due to the ongoing conflict between China and Taiwan.) The reason for choosing this Hung Shui Oolong is that it comes closest, in my opinion, to the ideal Dong Ding Oolong: the roasting hasn't burned the leaves and they still unfold very nicely. The aromas have been positively impacted by the roasting. There are scents of walnuts, chestnuts, but also fresh fruity smells. And the taste is both long and sweet. This batch is less roasted than typical competition Dong Ding Oolongs.
 The next tea on our list was the spring 2016 Hung Shui Oolong from Alishan. The purpose of this tea is to understand that Dong Ding Oolong has become so famous in the 1980s, that such teas were (and still are) produced all over Taiwan. That's why the name Hung Shui Oolong has come up to describe an Oolong that has been produced with a typical Dong Ding Oolong process. This one comes from Alishan, from a higher elevation than the Dong Ding area. It's also made from Qingxin Oolong leaves. Using high mountain leaves is a strategy used to make finer and more powerful Hung Shui Oolong. That's why most teas that win the Dong Ding Oolong competition don't come from Dong Ding, but from Shan Lin Xi or even Lishan! What's also interesting is that this Alishan Hung Shui Oolong has a stronger roast, which adds even more dark aromas to the brew.
The varying roasting level is what makes Oolong so complex and so fascinating. Other teas are not roasted, but simply dried. Thanks to the roast, new flavors appear. The brew become malty and reminds us of a brandy! This also has a big impact on how the leaves are best brewed. That's what Sarah and Rattana could learn and practice with me. (I explain the brewing in my TeaMasters guide to Brewing Oolong tea, which I offer for any order of 60 USD or more.)
It's SO good!
Then we tasted a very similar tea: this summer 2018 zhuo yan Oolong from Yong Lung in Dong Ding. Zhuo yan means jassid bitten. The difference with the first Hung Shui Oolong is not the roasting (this one also has a light to medium roast), but the season: summer. Because it's produced when the temperatures are higher, the oxidation level is slightly higher. The fragrances are much fruitier and sweet than in the spring or in winter. This is also a Dong Ding Oolong: it comes from Dong Ding and is processed like a Dong Ding Oolong, but the summer season has a tremendous impact on the brew.
2010 fall OB
We then proceeded to taste this 2010 fall Oriental Beauty Oolong from Hsin Chu. With this tea, I wanted to show the similarities and differences to the previous summer zhuo yan Oolong. There are 3 main differences: 1. the cultivar, qingxin dapang instead of qingxin Oolong, 2. the higher oxidation for Oriental Beauty and the shape and size of the dry leaves.

The zhuo yan Oolong from Yong Lung is like a bridge between Hung Shui Oolong and Oriental Beauty Oolong. So, tasting an Oriental Beauty just after the zhuo yan Oolong (or a concubine Oolong, which is not the same) helps to clarify the characteristics of each type of Oolong.

This was really a class, because we only used this ivory white porcelain gaiwan. An Yixing teapot could have given us a deeper and smoother taste, but the porcelain gaiwan brings more clarity and truth to the aromas.

Before I mention the next Oolong we brewed to conclude this lesson, I want to point out that these 2 and the following pictures were taken by Rattana with my Nikon D750. He's an accomplished professional photographer and I'm very grateful for the art he has created and his permission to share his photos here.
The most liked picture of 2018 is a black and white picture and these look stunning! The white porcelain enhances the purity of this tea experience!
It also makes sense to use black and white now, because we tasted this aged 1999 Hung Shui Oolong from Yong Lung (Dong Ding) as our last tea.
It takes us back 20 years ago. We notice that the leaves are mostly single, ie not attached to one another on a stem like nowadays. The look of dry leaves is already more refined!
Their aged aromas are very different from a newly roasted Hung Shui Oolong. It's difficult to describe. In wine, they are called tertiary aromas, scents that are not the result of a specific process, but of time. The taste is smoother, less powerful also.
But the character of Dong Ding remains the same.
Brew after brew and cup after cup.
I hope you all had a great start in 2019! I couldn't have dreamed a better one myself. Thank you Rattana for capturing the beauty of this dark dragon!

OUR VALUED CUSTOMERS: “All you gotta do is...”

Better Embedded System SW: Sometimes Bug Severity Isn't the Most Important Thing

Generally you need to take into account both the consequence of a software defect as well as how often it occurs when doing bug triage.  (See: Using a Risk Analysis Table to Categorize Bug Priority)

But an important special case is one in which the consequence is a business consequence such as brand tarnish rather than a spectacular software crash.   I used to use a hypothetical example of the audience's company name being misspelled on the system display to illustrate the point.  Well, it's not hypothetical any more!

Lamborghini sells a quarter-million dollar SUV with numerous software defects, including spelling the company name as "Lanborghini"   Guess which defect gets the press?


And it turns out that a software update not only didn't solve the typo, but also broke a bunch more functionality.  

OCaml Planet: How configurator reads C constants

Dune comes with a library to query OS-specific information, called configurator. It is able to evaluate C expressions and turn them into OCaml value. Surprisingly, it even works when compiling for a different architecture. How can it do that?

OUR VALUED CUSTOMERS: The holidays are a challenging time...


Which Lights?: left-reminders: Fuck:

left-reminders:

Fuck:

Trivium: 02jan2019

Daniel Lemire's blog: Memory-level parallelism: Intel Skylake versus Intel Cannonlake

All programmers know about multicore parallelism: your CPU is made of several nearly independent processors (called cores) that can run instructions in parallel. However, our processors are parallel in many different ways. I am interested in a particular form of parallelism called “memory-level parallelism” where the same processor can issue several memory requests. This is an important form of parallelism because current memory subsystems have high latency: it can take dozens of nanoseconds or more between the moment the processor asks for data and the time the data comes back from RAM. The general trend has not been a positive one in this respect: in many cases, the more advanced and expensive the processor, the higher the latency. To compensate for the high latency, we have parallelism: you can ask for many data elements from the memory subsystems at the same time.

In earlier work, we showed that current Intel processors (Skylake microarchitecture) are limited to about ten concurrent memory requests whereas Apple’s A12 processor scale to 40 or more memory requests.

Intel just released a more recent microarchitecture (cannonlake) and we have been putting it to the test. Is Intel improving?

It seems so. In a benchmark where you randomly access a large array, using a number of separate paths (which I call “lanes”), we find that the cannonlake processor appears to support twice as many concurrent memory requests as the skylake processors.

The Skylake processor has lower latency (70 ns/query) compared to the Cannonlake processor (110 ns/query). Nevertheless, the Cannonlake is eventually able to beat the Skylake processor in bandwidth by a wide margin (12 GB/s vs 9 GB/s).

The story is similar to the Apple A12 experiments.

This suggests that even though future processors may not have lower latency when accessing memory, we might be better able to hide this latency through more parallelism.

Even if you are writing single-threaded code, you ought to think more and more about parallelism.

Our code is available.

Credit: Though all the mistakes are mine, this is joint work with Travis Downs.

Further details: Processors access the memory through pages. By default, many Intel systems have “small” pages (4kB). When doing random accesses in large memory regions, you are likely to access too many pages, so that you incur expensive “page misses” that lead to “page walks”. It is possible to use large page sizes, even “huge pages”. But since memory is allocated in pages and you may end up with many under-utilized pages if they are too large. In practice, under-utilized pages (sometimes called “memory fragmentation) can be detrimental to performance. To get the good results above, I use huge pages. Because there is just one large memory allocation in my tests, memory fragmentation is not a concern. With small pages, the Cannonlake processor loses its edge over Skylake: they are both limited to about 9 concurrent requests. Thankfully, on Linux, programmers can request huge pages with a madvise call when they know it is a good idea.

OCaml Weekly News: OCaml Weekly News, 01 Jan 2019

  1. OCaml implementations of Raft consensus protocol
  2. Online, realtime, multiplayer Set game written in ocaml
  3. Is it safe to hold onto string data in C bindings?
  4. Other OCaml News

OCaml Planet: Using TLA+ to understand Xen vchan

The vchan protocol is used to stream data between virtual machines on a Xen host without needing any locks. It is largely undocumented. The TLA Toolbox is a set of tools for writing and checking specifications. In this post, I’ll describe my experiences using these tools to understand how the vchan protocol works.

Table of Contents

( this post also appeared on Reddit, Hacker News and Lobsters )

Background

Qubes and the vchan protocol

I run QubesOS on my laptop. A QubesOS desktop environment is made up of multiple virtual machines. A privileged VM, called dom0, provides the desktop environment and coordinates the other VMs. dom0 doesn’t have network access, so you have to use other VMs for doing actual work. For example, I use one VM for email and another for development work (these are called “application VMs”). There is another VM (called sys-net) that connects to the physical network, and yet another VM (sys-firewall) that connects the application VMs to net-vm.

My QubesOS desktop. The windows with blue borders are from my Debian development VM, while the green one is from a Fedora VM, etc.

The default sys-firewall is based on Fedora Linux. A few years ago, I replaced sys-firewall with a MirageOS unikernel. MirageOS is written in OCaml, and has very little C code (unlike Linux). It boots much faster and uses much less RAM than the Fedora-based VM. But recently, a user reported that restarting mirage-firewall was taking a very long time. The problem seemed to be that it was taking several minutes to transfer the information about the network configuration to the firewall. This is sent over vchan. The user reported that stracing the QubesDB process in dom0 revealed that it was sleeping for 10 seconds between sending the records, suggesting that a wakeup event was missing.

The lead developer of QubesOS said:

I’d guess missing evtchn trigger after reading/writing data in vchan.

Perhaps ocaml-vchan, the OCaml implementation of vchan, wasn’t implementing the vchan specification correctly? I wanted to check, but there was a problem: there was no vchan specification.

The Xen wiki lists vchan under Xen Document Days/TODO. The initial Git commit on 2011-10-06 said:

libvchan: interdomain communications library

This library implements a bidirectional communication interface between applications in different domains, similar to unix sockets. Data can be sent using the byte-oriented libvchan_read/libvchan_write or the packet-oriented libvchan_recv/libvchan_send.

Channel setup is done using a client-server model; domain IDs and a port number must be negotiated prior to initialization. The server allocates memory for the shared pages and determines the sizes of the communication rings (which may span multiple pages, although the default places rings and control within a single page).

With properly sized rings, testing has shown that this interface provides speed comparable to pipes within a single Linux domain; it is significantly faster than network-based communication.

I looked in the xen-devel mailing list around this period in case the reviewers had asked about how it worked.

One reviewer suggested:

Please could you say a few words about the functionality this new library enables and perhaps the design etc? In particular a protocol spec would be useful for anyone who wanted to reimplement for another guest OS etc. […] I think it would be appropriate to add protocol.txt at the same time as checking in the library.

However, the submitter pointed out that this was unnecessary, saying:

The comments in the shared header file explain the layout of the shared memory regions; any other parts of the protocol are application-defined.

Now, ordinarily, I wouldn’t be much interested in spending my free time tracking down race conditions in 3rd-party libraries for the benefit of strangers on the Internet. However, I did want to have another play with TLA…

TLA+

TLA+ is a language for specifying algorithms. It can be used for many things, but it is particularly designed for stateful parallel algorithms.

I learned about TLA while working at Docker. Docker EE provides software for managing large clusters of machines. It includes various orchestrators (SwarmKit, Kubernetes and Swarm Classic) and a web UI. Ensuring that everything works properly is very important, and to this end a large collection of tests had been produced. Part of my job was to run these tests. You take a test from a list in a web UI and click whatever buttons it tells you to click, wait for some period of time, and then check that what you see matches what the test says you should see. There were a lot of these tests, and they all had to be repeated on every supported platform, and for every release, release candidate or preview release. There was a lot of waiting involved and not much thinking required, so to keep my mind occupied, I started reading the TLA documentation.

I read The TLA+ Hyperbook and Specifying Systems. Both are by Leslie Lamport (the creator of TLA), and are freely available online. They’re both very easy to read. The hyperbook introduces the tools right away so you can start playing, while Specifying Systems starts with more theory and discusses the tools later. I think it’s worth reading both.

Once Docker EE 2.0 was released, we engineers were allowed to spend a week on whatever fun (Docker-related) project we wanted. I used the time to read the SwarmKit design documents and make a TLA model of that. I felt that using TLA prompted useful discussions with the SwarmKit developers (which can see seen in the pull request comments).

A specification document can answer questions such as:

  1. What does it do? (requirements / properties)
  2. How does it do it? (the algorithm)
  3. Does it work? (model checking)
  4. Why does it work? (inductive invariant)
  5. Does it really work? (proofs)

You don’t have to answer all of them to have a useful document, but I will try to answer each of them for vchan.

Is TLA useful?

In my (limited) experience with TLA, whenever I have reached the end of a specification (whether reading it or writing it), I always find myself thinking “Well, that was obvious. It hardly seems worth writing a spec for that!”. You might feel the same after reading this blog post.

To judge whether TLA is useful, I suggest you take a few minutes to look at the code. If you are good at reading C code then you might find, like the Xen reviewers, that it is quite obvious what it does, how it works, and why it is correct. Or, like me, you might find you’d prefer a little help. You might want to jot down some notes about it now, to see whether you learn anything new.

To give the big picture:

  1. Two VMs decide to communicate over vchan. One will be the server and the other the client.
  2. The server allocates three chunks of memory: one to hold data in transit from the client to the server, one for data going from server to client, and the third to track information about the state of the system. This includes counters saying how much data has been written and how much read, in each direction.
  3. The server tells Xen to grant the client access to this memory.
  4. The client asks Xen to map the memory into its address space. Now client and server can both access it at once. There are no locks in the protocol, so be careful!
  5. Either end sends data by writing it into the appropriate buffer and updating the appropriate counter in the shared block. The buffers are ring buffers, so after getting to the end, you start again from the beginning.
  6. The data-written (producer) counter and the data-read (consumer) counter together tell you how much data is in the buffer, and where it is. When the difference is zero, the reader must stop reading and wait for more data. When the difference is the size of the buffer, the writer must stop writing and wait for more space.
  7. When one end is waiting, the other can signal it using a Xen event channel. This essentially sets a pending flag to true at the other end, and wakes the VM if it is sleeping. If a VM tries to sleep while it has an event pending, it will immediately wake up again. Sending an event when one is already pending has no effect.

The public/io/libxenvchan.h header file provides some information, including the shared structures and comments about them:

xen/include/public/io/libxenvchan.h
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
struct ring_shared {
    uint32_t cons, prod;
};

#define VCHAN_NOTIFY_WRITE 0x1
#define VCHAN_NOTIFY_READ 0x2

/**
 * vchan_interface: primary shared data structure
 */
struct vchan_interface {
    /**
     * Standard consumer/producer interface, one pair per buffer
     * left is client write, server read
     * right is client read, server write
     */
    struct ring_shared left, right;
    /**
     * size of the rings, which determines their location
     * 10   - at offset 1024 in ring's page
     * 11   - at offset 2048 in ring's page
     * 12+  - uses 2^(N-12) grants to describe the multi-page ring
     * These should remain constant once the page is shared.
     * Only one of the two orders can be 10 (or 11).
     */
    uint16_t left_order, right_order;
    /**
     * Shutdown detection:
     *  0: client (or server) has exited
     *  1: client (or server) is connected
     *  2: client has not yet connected
     */
    uint8_t cli_live, srv_live;
    /**
     * Notification bits:
     *  VCHAN_NOTIFY_WRITE: send notify when data is written
     *  VCHAN_NOTIFY_READ: send notify when data is read (consumed)
     * cli_notify is used for the client to inform the server of its action
     */
    uint8_t cli_notify, srv_notify;
    /**
     * Grant list: ordering is left, right. Must not extend into actual ring
     * or grow beyond the end of the initial shared page.
     * These should remain constant once the page is shared, to allow
     * for possible remapping by a client that restarts.
     */
    uint32_t grants[0];
};

You might also like to look at the vchan source code. Note that the libxenvchan.h file in this directory includes and extends the above header file (with the same name).

For this blog post, we will ignore the Xen-specific business of sharing the memory and telling the client where it is, and assume that the client has mapped the memory and is ready to go.

Basic TLA concepts

We’ll take a first look at TLA concepts and notation using a simplified version of vchan. TLA comes with excellent documentation, so I won’t try to make this a full tutorial, but hopefully you will be able to follow the rest of this blog post after reading it. We will just consider a single direction of the channel (e.g. client-to-server) here.

Variables, states and behaviour

A variable in TLA is just what a programmer expects: something that changes over time. For example, I’ll use Buffer to represent the data currently being transmitted.

We can also add variables that are just useful for the specification. I use Sent to represent everything the sender-side application asked the vchan library to transmit, and Got for everything the receiving application has received:

1
VARIABLES Got, Buffer, Sent

A state in TLA represents a snapshot of the world at some point. It gives a value for each variable. For example, { Got: "H", Buffer: "i", Sent: "Hi", ... } is a state. The ... is just a reminder that a state also includes everything else in the world, not just the variables we care about.

Here are some more states:

State Got Buffer Sent
s0      
s1   H H
s2 H   H
s3 H i Hi
s4 Hi   Hi
s5 iH   Hi

A behaviour is a sequence of states, representing some possible history of the world. For example, << s0, s1, s2, s3, s4 >> is a behaviour. So is << s0, s1, s5 >>, but not one we want. The basic idea in TLA is to specify precisely which behaviours we want and which we don’t want.

A state expression is an expression that can be evaluated in the context of some state. For example, this defines Integrity to be a state expression that is true whenever what we have got so far matches what we wanted to send:

1
2
3
4
5
6
7
8
(* Take(m, i) is just the first i elements of message m. *)
Take(m, i) == SubSeq(m, 1, i)

(* Everything except the first i elements of message m. *)
Drop(m, i) == SubSeq(m, i + 1, Len(m))

Integrity ==
  Take(Sent, Len(Got)) = Got

Integrity is true for all the states above except for s5. I added some helper operators Take and Drop here. Sequences in TLA+ can be confusing because they are indexed from 1 rather than from 0, so it is easy to make off-by-one errors. These operators just use lengths, which we can all agree on. In Python syntax, it would be written something like:

1
2
def Integrity(s):
    return s.Sent.starts_with(s.Got)

A temporal formula is an expression that is evaluated in the context of a complete behaviour. It can use the temporal operators, which include:

  • [] (that’s supposed to look like a square) : “always”
  • <> (that’s supposed to look like a diamond) : “eventually”

[] F is true if the expression F is true at every point in the behaviour. <> F is true if the expression F is true at any point in the behaviour.

Messages we send should eventually arrive. Here’s one way to express that:

1
2
3
Availability ==
  \A x \in Nat :
    [] (Len(Sent) = x => <> (Len(Got) >= x) )

TLA syntax is a bit odd. It’s rather like LaTeX (which is not surprising: Lamport is also the “La” in LaTeX). \A means “for all” (rendered as an upside-down A). So this says that for every number x, it is always true that if we have sent x bytes then eventually we will have received at least x bytes.

This pattern of [] (F => <>G) is common enough that it has a shorter notation of F ~> G, which is read as “F (always) leads to G”. So, Availability can also be written as:

1
2
3
Availability ==
  \A x \in Nat :
    Len(Sent) = x ~> Len(Got) >= x

We’re only checking the lengths in Availability, but combined with Integrity that’s enough to ensure that we eventually receive what we want. So ideally, we’d like to ensure that every possible behaviour of the vchan library will satisfy the temporal formula Properties:

1
2
Properties ==
  Availability /\ []Integrity

That /\ is “and” by the way, and \/ is “or”. I did eventually start to be able to tell one from the other, though I still think && and || would be easier. In case I forget to explain some syntax, A Summary of TLA lists most of it.

Actions

It is hopefully easy to see that Properties defines properties we want. A user of vchan would be happy to see that these are things they can rely on. But they don’t provide much help to someone trying to implement vchan. For that, TLA provides another way to specify behaviours.

An action in TLA is an expression that is evaluated in the context of a pair of states, representing a single atomic step of the system. For example:

1
2
3
4
5
Read ==
  /\ Len(Buffer) > 0
  /\ Got' = Got \o Buffer
  /\ Buffer' = << >>
  /\ UNCHANGED Sent

The Read action is true of a step if that step transfers all the data from Buffer to Got. Unprimed variables (e.g. Buffer) refer to the current state and primed ones (e.g. Buffer') refer to the next state. There’s some more strange notation here too:

  • We’re using /\ to form a bulleted list here rather than as an infix operator. This is indentation-sensitive. TLA also supports \/ lists in the same way.
  • \o is sequence concatenation (+ in Python).
  • << >> is the empty sequence ([ ] in Python).
  • UNCHANGED Sent means Sent' = Sent.

In Python, it might look like this:

1
2
3
4
5
def Read(current, next):
  return Len(current.Buffer) > 0 \
     and next.Got = current.Got + current.Buffer \
     and next.Buffer = [] \
     and next.Sent = current.Sent

Actions correspond more closely to code than temporal formulas, because they only talk about how the next state is related to the current one.

This action only allows one thing: reading the whole buffer at once. In the C implementation of vchan the receiving application can provide a buffer of any size and the library will read at most enough bytes to fill the buffer. To model that, we will need a slightly more flexible version:

1
2
3
4
5
Read ==
  \E n \in 1..Len(Buffer) :
    /\ Got' = Got \o Take(Buffer, n)
    /\ Buffer' = Drop(Buffer, n)
    /\ UNCHANGED Sent

This says that a step is a Read step if there is any n (in the range 1 to the length of the buffer) such that we transferred n bytes from the buffer. \E means “there exists …”.

A Write action can be defined in a similar way:

1
2
3
4
5
6
7
8
9
CONSTANT BufferSize
Byte == 0..255

Write ==
  \E m \in Seq(Byte) \ {<< >>} :
    /\ Buffer' = Buffer \o m
    /\ Len(Buffer') <= BufferSize
    /\ Sent' = Sent \o m
    /\ UNCHANGED Got

A CONSTANT defines a parameter (input) of the specification (it’s constant in the sense that it doesn’t change between states). A Write operation adds some message m to the buffer, and also adds a copy of it to Sent so we can talk about what the system is doing. Seq(Byte) is the set of all possible sequences of bytes, and \ {<< >>} just excludes the empty sequence.

A step of the combined system is either a Read step or a Write step:

1
2
Next ==
  Read \/ Write

We also need to define what a valid starting state for the algorithm looks like:

1
2
3
4
Init ==
  /\ Sent = << >>
  /\ Buffer = << >>
  /\ Got = << >>

Finally, we can put all this together to get a temporal formula for the algorithm:

1
2
3
4
vars == << Got, Buffer, Sent >>

Spec ==
  Init /\ [][Next]_vars

Some more notation here:

  • [Next]_vars (that’s Next in brackets with a subscript vars) means Next \/ UNCHANGED vars.
  • Using Init (a state expression) in a temporal formula means it must be true for the first state of the behaviour.
  • [][Action]_vars means that [Action]_vars must be true for each step.

TLA syntax requires the _vars subscript here. This is because other things can be going on in the world beside our algorithm, so it must always be possible to take a step without our algorithm doing anything.

Spec defines behaviours just like Properties does, but in a way that makes it more obvious how to implement the protocol.

Correctness of Spec

Now we have definitions of Spec and Properties, it makes sense to check that every behaviour of Spec satisfies Properties. In Python terms, we want to check that all behaviours b satisfy this:

1
2
def SpecOK(b):
  return Spec(b) = False or Properties(b)

i.e. either b isn’t a behaviour that could result from the actions of our algorithm or, if it is, it satisfies Properties. In TLA notation, we write this as:

1
2
SpecOK ==
  Spec => Properties

It’s OK if a behaviour is allowed by Properties but not by Spec. For example, the behaviour which goes straight from Got="", Sent="" to Got="Hi", Sent="Hi" in one step meets our requirements, but it’s not a behaviour of Spec.

The real implementation may itself further restrict Spec. For example, consider the behaviour << s0, s1, s2 >>:

State Got Buffer Sent
s0   Hi Hi
s1 H i Hi
s2 Hi   Hi

The sender sends two bytes at once, but the reader reads them one at a time. This is a behaviour of the C implementation, because the reading application can ask the library to read into a 1-byte buffer. However, it is not a behaviour of the OCaml implementation, which gets to choose how much data to return to the application and will return both bytes together.

That’s fine. We just need to show that OCamlImpl => Spec and Spec => Properties and we can deduce that OCamlImpl => Properties. This is, of course, the key purpose of a specification: we only need to check that each implementation implements the specification, not that each implementation directly provides the desired properties.

It might seem strange that an implementation doesn’t have to allow all the specified behaviours. In fact, even the trivial specification Spec == FALSE is considered to be a correct implementation of Properties, because it has no bad behaviours (no behaviours at all). But that’s OK. Once the algorithm is running, it must have some behaviour, even if that behaviour is to do nothing. As the user of the library, you are responsible for checking that you can use it (e.g. by ensuring that the Init conditions are met). An algorithm without any behaviours corresponds to a library you could never use, not to one that goes wrong once it is running.

The model checker

Now comes the fun part: we can ask TLC (the TLA model checker) to check that Spec => Properties. You do this by asking the toolbox to create a new model (I called mine SpecOK) and setting Spec as the “behaviour spec”. It will prompt for a value for BufferSize. I used 2. There will be various things to fix up:

  • To check Write, TLC first tries to get every possible Seq(Byte), which is an infinite set. I defined MSG == Seq(Byte) and changed Write to use MSG. I then added an alternative definition for MSG in the model so that we only send messages of limited length. In fact, my replacement MSG ensures that Sent will always just be an incrementing sequence (<< 1, 2, 3, ... >>). That’s enough to check Properties, and much quicker than checking every possible message.
  • The system can keep sending forever. I added a state constraint to the model: Len(Sent) < 4 This tells TLC to stop considering any execution once this becomes false.

With that, the model runs successfully. This is a nice feature of TLA: instead of changing our specification to make it testable, we keep the specification correct and just override some aspects of it in the model. So, the specification says we can send any message, but the model only checks a few of them.

Now we can add Integrity as an invariant to check. That passes, but it’s good to double-check by changing the algorithm. I changed Read so that it doesn’t clear the buffer, using Buffer' = Drop(Buffer, 0) (with 0 instead of n). Then TLC reports a counter-example (“Invariant Integrity is violated”):

  1. The sender writes << 1, 2 >> to Buffer.
  2. The reader reads one byte, to give Got=1, Buffer=12, Sent=12.
  3. The reader reads another byte, to give Got=11, Buffer=12, Sent=12.

Looks like it really was checking what we wanted. It’s good to be careful. If we’d accidentally added Integrity as a “property” to check rather than as an “invariant” then it would have interpreted it as a temporal formula and reported success just because it is true in the initial state.

One really nice feature of TLC is that (unlike a fuzz tester) it does a breadth-first search and therefore finds minimal counter-examples for invariants. The example above is therefore the quickest way to violate Integrity.

Checking Availability complains because of the use of Nat (we’re asking it to check for every possible length). I replaced the Nat with AvailabilityNat and overrode that to be 0..4 in the model. It then complains “Temporal properties were violated” and shows an example where the sender wrote some data and the reader never read it.

The problem is, [Next]_vars always allows us to do nothing. To fix this, we can specify a “weak fairness” constraint. WF_vars(action), says that we can’t just stop forever with action being always possible but never happening. I updated Spec to require the Read action to be fair:

1
Spec == Init /\ [][Next]_vars /\ WF_vars(Read)

Again, care is needed here. If we had specified WF_vars(Next) then we would be forcing the sender to keep sending forever, which users of vchan are not required to do. Worse, this would mean that every possible behaviour of the system would result in Sent growing forever. Every behaviour would therefore hit our Len(Sent) < 4 constraint and TLC wouldn’t consider it further. That means that TLC would never check any actual behaviour against Availability, and its reports of success would be meaningless! Changing Read to require n \in 2..Len(Buffer) is a quick way to see that TLC is actually checking Availability.

Here’s the complete spec so far: vchan1.pdf (source)

The real vchan

The simple Spec algorithm above has some limitations. One obvious simplification is that Buffer is just the sequence of bytes in transit, whereas in the real system it is a ring buffer, made up of an array of bytes along with the producer and consumer counters. We could replace it with three separate variables to make that explicit. However, ring buffers in Xen are well understood and I don’t feel that it would make the specification any clearer to include that.

A more serious problem is that Spec assumes that there is a way to perform the Read and Write operations atomically. Otherwise the real system would have behaviours not covered by the spec. To implement the above Spec correctly, you’d need some kind of lock. The real vchan protocol is more complicated than Spec, but avoids the need for a lock.

The real system has more shared state than just Buffer. I added extra variables to the spec for each item of shared state in the C code, along with its initial value:

  • SenderLive = TRUE (sender sets to FALSE to close connection)
  • ReceiverLive = TRUE (receiver sets to FALSE to close connection)
  • NotifyWrite = TRUE (receiver wants to be notified of next write)
  • DataReadyInt = FALSE (sender has signalled receiver over event channel)
  • NotifyRead = FALSE (sender wants to be notified of next read)
  • SpaceAvailableInt = FALSE (receiver has notified sender over event channel)

DataReadyInt represents the state of the receiver’s event port. The sender can make a Xen hypercall to set this and wake (or interrupt) the receiver. I guess sending these events is somewhat slow, because the NotifyWrite system is used to avoid sending events unnecessarily. Likewise, SpaceAvailableInt is the sender’s event port.

The algorithm

Here is my understanding of the protocol. On the sending side:

  1. The sending application asks to send some bytes.
    We check whether the receiver has closed the channel and abort if so.
  2. We check the amount of buffer space available.
  3. If there isn’t enough, we set NotifyRead so the receiver will notify us when there is more.
    We also check the space again after this, in case it changed while setting the flag.
  4. If there is any space:
    • We write as much data as we can to the buffer.
    • If the NotifyWrite flag is set, we clear it and notify the receiver of the write.
  5. If we wrote everything, we return success.
  6. Otherwise, we wait to be notified of more space.
  7. We check whether the receiver has closed the channel.
    If so we abort. Otherwise, we go back to step 2.

On the receiving side:

  1. The receiving application asks us to read up to some amount of data.
  2. We check the amount of data available in the buffer.
  3. If there isn’t as much as requested, we set NotifyWrite so the sender will notify us when there is.
    We also check the space again after this, in case it changed while setting the flag.
  4. If there is any data, we read up to the amount requested.
    If the NotifyRead flag is set, we clear it and notify the sender of the new space.
    We return success to the application (even if we didn’t get as much as requested).
  5. Otherwise (if there was no data), we check whether the sender has closed the connection.
  6. If not (if the connection is still open), we wait to be notified of more data, and then go back to step 2.

Either side can close the connection by clearing their “live” flag and signalling the other side. I assumed there is also some process-local way that the close operation can notify its own side if it’s currently blocked.

To make expressing this kind of step-by-step algorithm easier, TLA+ provides a programming-language-like syntax called PlusCal. It then translates PlusCal into TLA actions.

Confusingly, there are two different syntaxes for PlusCal: Pascal style and C style. This means that, when you search for examples on the web, there is a 50% chance they won’t work because they’re using the other flavour. I started with the Pascal one because that was the first example I found, but switched to C-style later because it was more compact.

Here is my attempt at describing the sender algorithm above in PlusCal:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
  fair process (SenderWrite = SenderWriteID)
  variables free = 0,     \* Our idea of how much free space is available.
            msg = << >>,  \* The data we haven't sent yet.
            Sent = << >>; \* Everything we were asked to send.
  {
sender_ready:-        while (TRUE) {
                        if (~SenderLive \/ ~ReceiverLive) goto Done
                        else {
                          with (m \in MSG) { msg := m };
                          Sent := Sent \o msg;    \* Remember we wanted to send this
                        };
sender_write:           while (TRUE) {
                          free := BufferSize - Len(Buffer);
sender_request_notify:    if (free >= Len(msg)) goto sender_write_data
                          else NotifyRead := TRUE;
sender_recheck_len:       free := BufferSize - Len(Buffer);
sender_write_data:        if (free > 0) {
                            Buffer := Buffer \o Take(msg, Min(Len(msg), free));
                            msg := Drop(msg, Min(Len(msg), free));
                            free := 0;
sender_check_notify_data:   if (NotifyWrite) {
                              NotifyWrite := FALSE;   \* Atomic test-and-clear
sender_notify_data:           DataReadyInt := TRUE;   \* Signal receiver
                              if (msg = << >>) goto sender_ready
                            } else if (msg = << >>) goto sender_ready
                          };
sender_blocked:           await SpaceAvailableInt \/ ~SenderLive;
                          if (~SenderLive) goto Done;
                          else SpaceAvailableInt := FALSE;
sender_check_recv_live:   if (~ReceiverLive) goto Done;
                        }
                      }
  }

The labels (e.g. sender_request_notify:) represent points in the program where other actions can happen. Everything between two labels is considered to be atomic. I checked that every block of code between labels accesses only one shared variable. This means that the real system can’t see any states that we don’t consider. The toolbox doesn’t provide any help with this; you just have to check manually.

The sender_ready label represents a state where the client application hasn’t yet decided to send any data. Its label is tagged with - to indicate that fairness doesn’t apply here, because the protocol doesn’t require applications to keep sending more data forever. The other steps are fair, because once we’ve decided to send something we should keep going.

Taking a step from sender_ready to sender_write corresponds to the vchan library’s write function being called with some argument m. The with (m \in MSG) says that m could be any message from the set MSG. TLA also contains a CHOOSE operator that looks like it might do the same thing, but it doesn’t. When you use with, you are saying that TLC should check all possible messages. When you use CHOOSE, you are saying that it doesn’t matter which message TLC tries (and it will always try the same one). Or, in terms of the specification, a CHOOSE would say that applications can only ever send one particular message, without telling you what that message is.

In sender_write_data, we set free := 0 for no obvious reason. This is just to reduce the number of states that the model checker needs to explore, since we don’t care about its value after this point.

Some of the code is a little awkward because I had to put things in else branches that would more naturally go after the whole if block, but the translator wouldn’t let me do that. The use of semi-colons is also a bit confusing: the PlusCal-to-TLA translator requires them after a closing brace in some places, but the PDF generator messes up the indentation if you include them.

Here’s how the code block starting at sender_request_notify gets translated into a TLA action:

1
2
3
4
5
6
7
8
9
10
11
sender_request_notify ==
  /\ pc[SenderWriteID] = "sender_request_notify"
  /\ IF free >= Len(msg)
        THEN /\ pc' = [pc EXCEPT ![SenderWriteID] = "sender_write_data"]
             /\ UNCHANGED NotifyRead
        ELSE /\ NotifyRead' = TRUE
             /\ pc' = [pc EXCEPT ![SenderWriteID] = "sender_recheck_len"]
  /\ UNCHANGED << SenderLive, ReceiverLive, Buffer,
                  NotifyWrite, DataReadyInt,
                  SpaceAvailableInt, free, msg, Sent,
                  have, want, Got >>

pc is a mapping from process ID to the label where that process is currently executing. So sender_request_notify can only be performed when the SenderWriteID process is at the sender_request_notify label. Afterwards pc[SenderWriteID] will either be at sender_write_data or sender_recheck_len (if there wasn’t enough space for the whole message).

Here’s the code for the receiver:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
  fair process (ReceiverRead = ReceiverReadID)
  variables have = 0,     \* The amount of data we think the buffer contains.
            want = 0,     \* The amount of data the user wants us to read.
            Got = << >>;  \* Pseudo-variable recording all data ever received by receiver.
  {
recv_ready:         while (ReceiverLive) {
                      with (n \in 1..MaxReadLen) want := n;
recv_reading:         while (TRUE) {
                        have := Len(Buffer);
recv_got_len:           if (have >= want) goto recv_read_data
                        else NotifyWrite := TRUE;
recv_recheck_len:       have := Len(Buffer);
recv_read_data:         if (have > 0) {
                          Got := Got \o Take(Buffer, Min(want, have));
                          Buffer := Drop(Buffer, Min(want, have));
                          want := 0;
                          have := 0;
recv_check_notify_read:   if (NotifyRead) {
                            NotifyRead := FALSE;      \* (atomic test-and-clear)
recv_notify_read:           SpaceAvailableInt := TRUE;
                            goto recv_ready;          \* Return success
                          } else goto recv_ready;     \* Return success
                        } else if (~SenderLive \/ ~ReceiverLive) {
                          goto Done;
                        };
recv_await_data:        await DataReadyInt \/ ~ReceiverLive;
                        if (~ReceiverLive) { want := 0; goto Done }
                        else DataReadyInt := FALSE;
                      }
                    }
  }

It’s quite similar to before. recv_ready corresponds to a state where the application hasn’t yet called read. When it does, we take n (the maximum number of bytes to read) as an argument and store it in the local variable want.

Note: you can use the C library in blocking or non-blocking mode. In blocking mode, a write (or read) waits until data is sent (or received). In non-blocking mode, it returns a special code to the application indicating that it needs to wait. The application then does the waiting itself and then calls the library again. I think the specification above covers both cases, depending on whether you think of sender_blocked and recv_await_data as representing code inside or outside of the library.

We also need a way to close the channel. It wasn’t clear to me, from looking at the C headers, when exactly you’re allowed to do that. I think that if you had a multi-threaded program and you called the close function while the write function was blocked, it would unblock and return. But if you happened to call it at the wrong time, it would try to use a closed file descriptor and fail (or read from the wrong one). So I guess it’s single threaded, and you should use the non-blocking mode if you want to cancel things.

That means that the sender can close only when it is at sender_ready or sender_blocked, and similarly for the receiver. The situation with the OCaml code is the same, because it is cooperatively threaded and so the close operation can only be called while blocked or idle. However, I decided to make the specification more general and allow for closing at any point by modelling closing as separate processes:

1
2
3
4
5
6
7
8
9
  fair process (SenderClose = SenderCloseID) {
    sender_open:-         SenderLive := FALSE;  \* Clear liveness flag
    sender_notify_closed: DataReadyInt := TRUE; \* Signal receiver
  }

  fair process (ReceiverClose = ReceiverCloseID) {
    recv_open:-         ReceiverLive := FALSE;      \* Clear liveness flag
    recv_notify_closed: SpaceAvailableInt := TRUE;  \* Signal sender
  }

Again, the processes are “fair” because once we start closing we should finish, but the initial labels are tagged with “-“ to disable fairness there: it’s OK if you keep a vchan open forever.

There’s a slight naming problem here. The PlusCal translator names the actions it generates after the starting state of the action. So sender_open is the action that moves from the sender_open label. That is, the sender_open action actually closes the connection!

Finally, we share the event channel with the buffer going in the other direction, so we might get notifications that are nothing to do with us. To ensure we handle that, I added another process that can send events at any time:

1
2
3
4
5
6
  process (SpuriousInterrupts = SpuriousID) {
    spurious: while (TRUE) {
                either SpaceAvailableInt := TRUE
                or     DataReadyInt := TRUE
              }
  }

either/or says that we need to consider both possibilities. This process isn’t marked fair, because we can’t rely these interrupts coming. But we do have to handle them when they happen.

Testing the full spec

PlusCal code is written in a specially-formatted comment block, and you have to press Ctrl-T to generate (or update) then TLA translation before running the model checker.

Be aware that the TLA Toolbox is a bit unreliable about keyboard short-cuts. While typing into the editor always works, short-cuts such as Ctrl-S (save) sometimes get disconnected. So you think you’re doing “edit/save/translate/save/check” cycles, but really you’re just checking some old version over and over again. You can avoid this by always running the model checker with the keyboard shortcut too, since that always seems to fail at the same time as the others. Focussing a different part of the GUI and then clicking back in the editor again fixes everything for a while.

Anyway, running our model on the new spec shows that Integrity is still OK. However, the Availability check fails with the following counter-example:

  1. The sender writes << 1 >> to Buffer.
  2. The sender closes the connection.
  3. The receiver closes the connection.
  4. All processes come to a stop, but the data never arrived.

We need to update Availability to consider the effects of closing connections. And at this point, I’m very unsure what vchan is intended to do. We could say:

1
2
3
4
5
Availability ==
  \A x \in AvailabilityNat :
    Len(Sent) = x ~> \/ Len(Got) >= x
                     \/ ~ReceiverLive
                     \/ ~SenderLive

That passes. But vchan describes itself as being like a Unix socket. If you write to a Unix socket and then close it, you still expect the data to be delivered. So actually I tried this:

1
2
3
4
5
Availability ==
  \A x \in AvailabilityNat :
    x = Len(Sent) /\ SenderLive /\ pc[SenderWriteID] = "sender_ready" ~>
         \/ Len(Got) >= x
         \/ ~ReceiverLive

This says that if a sender write operation completes successfully (we’re back at sender_ready) and at that point the sender hasn’t closed the connection, then the receiver will eventually receive the data (or close its end).

That is how I would expect it to behave. But TLC reports that the new spec does not satisfy this, giving this example (simplified - there are 16 steps in total):

  1. The receiver starts reading. It finds that the buffer is empty.
  2. The sender writes some data to Buffer and returns to sender_ready.
  3. The sender closes the channel.
  4. The receiver sees that the connection is closed and stops.

Is this a bug? Without a specification, it’s impossible to say. Maybe vchan was never intended to ensure delivery once the sender has closed its end. But this case only happens if you’re very unlucky about the scheduling. If the receiving application calls read when the sender has closed the connection but there is data available then the C code does return the data in that case. It’s only if the sender happens to close the connection just after the receiver has checked the buffer and just before it checks the close flag that this happens.

It’s also easy to fix. I changed the code in the receiver to do a final check on the buffer before giving up:

1
2
3
4
                        } else if (~SenderLive \/ ~ReceiverLive) {
recv_final_check:         if (Len(Buffer) = 0) { want := 0; goto Done }
                          else goto recv_reading;
                        }

With that change, we can be sure that data sent while the connection is open will always be delivered (provided only that the receiver doesn’t close the connection itself). If you spotted this issue yourself while you were reviewing the code earlier, then well done!

Note that when TLC finds a problem with a temporal property (such as Availability), it does not necessarily find the shortest example first. I changed the limit on Sent to Len(Sent) < 2 and added an action constraint of ~SpuriousInterrupts to get a simpler example, with only 1 byte being sent and no spurious interrupts.

Some odd things

I noticed a couple of other odd things, which I thought I’d mention.

First, NotifyWrite is initialised to TRUE, which seemed unnecessary. We can initialise it to FALSE instead and everything still works. We can even initialise it with NotifyWrite \in {TRUE, FALSE} to allow either behaviour, and thus test that old programs that followed the original version of the spec still work with either behaviour.

That’s a nice advantage of using a specification language. Saying “the code is the spec” becomes less useful as you build up more and more versions of the code!

However, because there was no spec before, we can’t be sure that existing programs do follow it. And, in fact, I found that QubesDB uses the vchan library in a different and unexpected way. Instead of calling read, and then waiting if libvchan says to, QubesDB blocks first in all cases, and then calls the read function once it gets an event.

We can document that by adding an extra step at the start of ReceiverRead:

1
2
3
4
5
recv_init:          either goto recv_ready        \* (recommended)
                    or {    \* (QubesDB does this)
                      with (n \in 1..MaxReadLen) want := n;
                      goto recv_await_data;
                    };

Then TLC shows that NotifyWrite cannot start as FALSE.

The second odd thing is that the receiver sets NotifyRead whenever there isn’t enough data available to fill the application’s buffer completely. But usually when you do a read operation you just provide a buffer large enough for the largest likely message. It would probably make more sense to set NotifyWrite only when the buffer is completely empty. After checking the current version of the algorithm, I changed the specification to allow either behaviour.

Why does vchan work?

At this point, we have specified what vchan should do and how it does it. We have also checked that it does do this, at least for messages up to 3 bytes long with a buffer size of 2. That doesn’t sound like much, but we still checked 79,288 distinct states, with behaviours up to 38 steps long. This would be a perfectly reasonable place to declare the specification (and blog post) finished.

However, TLA has some other interesting abilities. In particular, it provides a very interesting technique to help discover why the algorithm works.

We’ll start with Integrity. We would like to argue as follows:

  1. Integrity is true in any initial state (i.e. Init => Integrity).
  2. Any Next step preserves Integrity (i.e. Integrity /\ Next => Integrity').

Then it would just be a matter looking at each possible action that makes up Next and checking that each one individually preserves Integrity. However, we can’t do this with Integrity because (2) isn’t true. For example, the state { Got: "", Buffer: "21", Sent: "12" } satisfies Integrity, but if we take a read step then the new state won’t. Instead, we have to argue “If we take a Next step in any reachable state then Integrity'”, but that’s very difficult because how do we know whether a state is reachable without searching them all?

So the idea is to make a stronger version of Integrity, called IntegrityI, which does what we want. IntegrityI is called an inductive invariant. The first step is fairly obvious - I began with:

1
2
IntegrityI ==
  Sent = Got \o Buffer \o msg

Integrity just said that Got is a prefix of Sent. This says specifically that the rest is Buffer \o msg - the data currently being transmitted and the data yet to be transmitted.

We can ask TLC to check Init /\ [][Next]_vars => []IntegrityI to check that it is an invariant, as before. It does that by finding all the Init states and then taking Next steps to find all reachable states. But we can also ask it to check IntegrityI /\ [][Next]_vars => []IntegrityI. That is, the same thing but starting from any state matching IntegrityI instead of Init.

I created a new model (IntegrityI) to do that. It reports a few technical problems at the start because it doesn’t know the types of anything. For example, it can’t choose initial values for SenderLive without knowing that SenderLive is a boolean. I added a TypeOK state expression that gives the expected type of every variable:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
MESSAGE == Seq(Byte)
FINITE_MESSAGE(L) == UNION ( { [ 1..N -> Byte ] : N \in 0..L } )

TypeOK ==
  /\ Sent \in MESSAGE
  /\ Got \in MESSAGE
  /\ Buffer \in FINITE_MESSAGE(BufferSize)
  /\ SenderLive \in BOOLEAN
  /\ ReceiverLive \in BOOLEAN
  /\ NotifyWrite \in BOOLEAN
  /\ DataReadyInt \in BOOLEAN
  /\ NotifyRead \in BOOLEAN
  /\ SpaceAvailableInt \in BOOLEAN
  /\ free \in 0..BufferSize
  /\ msg \in FINITE_MESSAGE(MaxWriteLen)
  /\ want \in 0..MaxReadLen
  /\ have \in 0..BufferSize

We also need to tell it all the possible states of pc (which says which label each process it at):

1
2
3
4
5
6
7
8
9
10
11
PCOK == pc \in [
    SW: {"sender_ready", "sender_write", "sender_request_notify", "sender_recheck_len",
         "sender_write_data", "sender_blocked", "sender_check_notify_data",
         "sender_notify_data", "sender_check_recv_live", "Done"},
    SC: {"sender_open", "sender_notify_closed", "Done"},
    RR: {"recv_init", "recv_ready", "recv_reading", "recv_got_len", "recv_recheck_len",
         "recv_read_data", "recv_final_check", "recv_await_data",
         "recv_check_notify_read", "recv_notify_read", "Done"},
    RC: {"recv_open", "recv_notify_closed", "Done"},
    SP: {"spurious"}
]

You might imagine that the PlusCal translator would generate that for you, but it doesn’t. We also need to override MESSAGE with FINITE_MESSAGE(n) for some n (I used 2). Otherwise, it can’t enumerate all possible messages. Now we have:

1
2
3
4
IntegrityI ==
  /\ TypeOK
  /\ PCOK
  /\ Sent = Got \o Buffer \o msg

With that out of the way, TLC starts finding real problems (that is, examples showing that IntegrityI /\ Next => IntegrityI' isn’t true). First, recv_read_data would do an out-of-bounds read if have = 1 and Buffer = << >>. Our job is to explain why that isn’t a valid state. We can fix it with an extra constraint:

1
2
3
4
5
IntegrityI ==
  /\ TypeOK
  /\ PCOK
  /\ Sent = Got \o Buffer \o msg
  /\ pc[ReceiverReadID] = "recv_read_data" => have <= Len(Buffer)

(note: that => is “implies”, while the <= is “less-than-or-equal-to”)

Now it complains that if we do recv_got_len with Buffer = << >>, have = 1, want = 0 then we end up in recv_read_data with Buffer = << >>, have = 1, and we have to explain why that can’t happen and so on.

Because TLC searches breadth-first, the examples it finds never have more than 2 states. You just have to explain why the first state can’t happen in the real system. Eventually, you get a big ugly pile of constraints, which you then think about for a bit and simply. I ended up with:

1
2
3
4
5
6
7
8
9
10
IntegrityI ==
  /\ TypeOK
  /\ PCOK
  /\ Sent = Got \o Buffer \o msg
  /\ have <= Len(Buffer)
  /\ free <= BufferSize - Len(Buffer)
  /\ pc[SenderWriteID] \in {"sender_write", "sender_request_notify", "sender_recheck_len",
                            "sender_write_data", "sender_blocked", "sender_check_recv_live"}
     => msg /= << >>
  /\ pc[SenderWriteID] \in {"sender_ready"} => msg = << >>

It’s a good idea to check the final IntegrityI with the original SpecOK model, just to check it really is an invariant.

So, in summary, Integrity is always true because:

  • Sent is always the concatenation of Got, Buffer and msg. That’s fairly obvious, because sender_ready sets msg and appends the same thing to Sent, and the other steps (sender_write_data and recv_read_data) just transfer some bytes from the start of one variable to the end of another.

  • Although, like all local information, the receiver’s have variable might be out-of-date, there must be at least that much data in the buffer, because the sender process will only have added more, not removed any. This is sufficient to ensure that we never do an out-of-range read.

  • Likewise, the sender’s free variable is a lower bound on the true amount of free space, because the receiver only ever creates more space. We will therefore never write beyond the free space.

I think this ability to explain why an algorithm works, by being shown examples where the inductive property doesn’t hold, is a really nice feature of TLA. Inductive invariants are useful as a first step towards writing a proof, but I think they’re valuable even on their own. If you’re documenting your own algorithm, this process will get you to explain your own reasons for believing it works (I tried it on a simple algorithm in my own code and it seemed helpful).

Some notes:

  • Originally, I had the free and have constraints depending on pc. However, the algorithm sets them to zero when not in use so it turns out they’re always true.

  • IntegrityI matches 532,224 states, even with a maximum Sent length of 1, but it passes! There are some games you can play to speed things up; see Using TLC to Check Inductive Invariance for some suggestions (I only discovered that while writing this up).

Proving Integrity

TLA provides a syntax for writing proofs, and integrates with TLAPS (the TLA+ Proof System) to allow them to be checked automatically.

Proving IntegrityI is just a matter of showing that Init => IntegrityI and that it is preserved by any possible [Next]_vars step. To do that, we consider each action of Next individually, which is long but simple enough.

I was able to prove it, but the recv_read_data action was a little difficult because we don’t know that want > 0 at that point, so we have to do some extra work to prove that transferring 0 bytes works, even though the real system never does that.

I therefore added an extra condition to IntegrityI that want is non-zero whenever it’s in use, and also conditions about have and free being 0 when not in use, for completeness:

1
2
3
4
5
6
7
8
9
10
IntegrityI ==
  [...]
  /\ want = 0 <=> pc[ReceiverReadID] \in {"recv_check_notify_read", "recv_notify_read",
                                          "recv_init", "recv_ready", "recv_notify_read",
                                          "Done"}
  /\ \/ pc[ReceiverReadID] \in {"recv_got_len", "recv_recheck_len", "recv_read_data"}
     \/ have = 0
  /\ \/ pc[SenderWriteID] \in {"sender_write", "sender_request_notify",
                               "sender_recheck_len", "sender_write_data"}
     \/ free = 0

Availability

Integrity was quite easy to prove, but I had more trouble trying to explain Availability. One way to start would be to add Availability as a property to check to the IntegrityI model. However, it takes a while to check properties as it does them at the end, and the examples it finds may have several steps (it took 1m15s to find a counter-example for me).

Here’s a faster way (37s). The algorithm will deadlock if both sender and receiver are in their blocked states and neither interrupt is pending, so I made a new invariant, I, which says that deadlock can’t happen:

1
2
3
4
5
6
I ==
  /\ IntegrityI
  /\ ~ /\ pc[SenderWriteID] = "sender_blocked"
       /\ ~SpaceAvailableInt
       /\ pc[ReceiverReadID] = "recv_await_data"
       /\ ~DataReadyInt

I discovered some obvious facts about closing the connection. For example, the SenderLive flag is set if and only if the sender’s close thread hasn’t done anything. I’ve put them all together in CloseOK:

1
2
3
4
5
6
7
8
9
10
11
12
(* Some obvious facts about shutting down connections. *)
CloseOK ==
  \* An endpoint is live iff its close thread hasn't done anything:
  /\ pc[SenderCloseID] = "sender_open" <=> SenderLive
  /\ pc[ReceiverCloseID] = "recv_open" <=> ReceiverLive
  \* The send and receive loops don't terminate unless someone has closed the connection:
  /\ pc[ReceiverReadID] \in {"recv_final_check", "Done"} => ~ReceiverLive \/ ~SenderLive
  /\ pc[SenderWriteID] \in {"Done"} => ~ReceiverLive \/ ~SenderLive
  \* If the receiver closed the connection then we will get (or have got) the signal:
  /\ pc[ReceiverCloseID] = "Done" =>
          \/ SpaceAvailableInt
          \/ pc[SenderWriteID] \in {"sender_check_recv_live", "Done"}

But I had problems with other examples TLC showed me, and I realised that I didn’t actually know why this algorithm doesn’t deadlock.

Intuitively it seems clear enough: the sender puts data in the buffer when there’s space and notifies the receiver, and the receiver reads it and notifies the writer. What could go wrong? But both processes are working with information that can be out-of-date. By the time the sender decides to block because the buffer looked full, the buffer might be empty. And by the time the receiver decides to block because it looked empty, it might be full.

Maybe you already saw why it works from the C code, or the algorithm above, but it took me a while to figure it out! I eventually ended up with an invariant of the form:

1
2
3
4
I ==
  ..
  /\ SendMayBlock    => SpaceWakeupComing
  /\ ReceiveMayBlock => DataWakeupComing

SendMayBlock is TRUE if we’re in a state that may lead to being blocked without checking the buffer’s free space again. Likewise, ReceiveMayBlock indicates that the receiver might block. SpaceWakeupComing and DataWakeupComing predict whether we’re going to get an interrupt. The idea is that if we’re going to block, we need to be sure we’ll be woken up. It’s a bit ugly, though, e.g.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
DataWakeupComing ==
  \/ DataReadyInt \* Event sent
  \/ pc[SenderWriteID] = "sender_notify_data"     \* Event being sent
  \/ pc[SenderCloseID] = "sender_notify_closed"
  \/ pc[ReceiverCloseID] = "recv_notify_closed"
  \/ /\ NotifyWrite   \* Event requested and ...
     /\ ReceiverLive  \* Sender can see receiver is still alive and ...
     /\ \/ pc[SenderWriteID] = "sender_write_data" /\ free > 0
        \/ pc[SenderWriteID] = "sender_check_notify_data"
        \/ pc[SenderWriteID] = "sender_recheck_len" /\ Len(Buffer) < BufferSize
        \/ pc[SenderWriteID] = "sender_ready" /\ SenderLive /\ Len(Buffer) < BufferSize
        \/ pc[SenderWriteID] = "sender_write" /\ Len(Buffer) < BufferSize
        \/ pc[SenderWriteID] = "sender_request_notify" /\ Len(Buffer) < BufferSize
        \/ SpaceWakeupComing /\ Len(Buffer) < BufferSize /\ SenderLive

It did pass my model that tested sending one byte, and I decided to try a proof. Well, it didn’t work. The problem seems to be that DataWakeupComing and SpaceWakeupComing are really mutually recursive. The reader will wake up if the sender wakes it, but the sender might be blocked, or about to block. That’s OK though, as long as the receiver will wake it, which it will do, once the sender wakes it…

You’ve probably already figured it out, but I thought I’d document my confusion. It occurred to me that although each process might have out-of-date information, that could be fine as long as at any one moment one of them was right. The last process to update the buffer must know how full it is, so one of them must have correct information at any given time, and that should be enough to avoid deadlock.

That didn’t work either. When you’re at a proof step and can’t see why it’s correct, you can ask TLC to show you an example. e.g. if you’re stuck trying to prove that sender_request_notify preserves I when the receiver is at recv_ready, the buffer is full, and ReceiverLive = FALSE, you can ask for an example of that:

1
2
3
4
5
6
7
Example ==
  /\ PCOK
  /\ pc[SenderWriteID] = "sender_request_notify"
  /\ pc[ReceiverReadID] = "recv_ready"
  /\ ReceiverLive = FALSE
  /\ I
  /\ Len(Buffer) = BufferSize

You then create a new model that searches Example /\ [][Next]_vars and tests I. As long as Example has several constraints, you can use a much larger model for this. I also ask it to check the property [][FALSE]_vars, which means it will show any step starting from Example.

It quickly became clear what was wrong: it is quite possible that neither process is up-to-date. If both processes see the buffer contains X bytes of data, and the sender sends Y bytes and the receiver reads Z bytes, then the sender will think there are X + Y bytes in the buffer and the receiver will think there are X - Z bytes, and neither is correct. My original 1-byte buffer was just too small to find a counter-example.

The real reason why vchan works is actually rather obvious. I don’t know why I didn’t see it earlier. But eventually it occurred to me that I could make use of Got and Sent. I defined WriteLimit to be the total number of bytes that the sender would write before blocking, if the receiver never did anything further. And I defined ReadLimit to be the total number of bytes that the receiver would read if the sender never did anything else.

Did I define these limits correctly? It’s easy to ask TLC to check some extra properties while it’s running. For example, I used this to check that ReadLimit behaves sensibly:

1
2
3
4
5
6
7
8
9
ReadLimitCorrect ==
  \* We will eventually receive what ReadLimit promises:
  /\ WF_vars(ReceiverRead) =>
      \A i \in AvailabilityNat :
        ReadLimit = i ~> Len(Got) >= i \/ ~ReceiverLive
  \* ReadLimit can only decrease if we decide to shut down:
  /\ [][ReadLimit' >= ReadLimit \/ ~ReceiverLive]_vars
  \* ReceiverRead steps don't change the read limit:
  /\ [][ReceiverRead => UNCHANGED ReadLimit \/ ~ReceiverLive]_vars

Because ReadLimit is defined in terms of what it does when no other processes run, this property should ideally be tested in a model without the fairness conditions (i.e. just Init /\ [][Next]_vars). Otherwise, fairness may force the sender to perform a step. We still want to allow other steps, though, to show that ReadLimit is a lower bound.

With this, we can argue that e.g. a 2-byte buffer will eventually transfer 3 bytes:

  1. The receiver will eventually read 3 bytes as long as the sender eventually sends 3 bytes.
  2. The sender will eventually send 3, if the receiver reads at least 1.
  3. The receiver will read 1 if the sender sends at least 1.
  4. The sender will send 1 if the reader has read at least 0 bytes, which is always true.

By this point, I was learning to be more cautious before trying a proof, so I added some new models to check this idea further. One prevents the sender from ever closing the connection and the other prevents the receiver from ever closing. That reduces the number of states to consider and I was able to check a slightly larger model.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
I ==
  /\ IntegrityI
  /\ CloseOK
  \* If the reader is stuck, but data is available, the sender will unblock it:
  /\ ReaderShouldBeUnblocked
     => \* The sender is going to write more:
        \/ WriteLimit > Len(Got) + Len(Buffer) /\ Len(msg) > 0 /\ SenderLive
        \* The sender is about to increase ReadLimit:
        \/ (\/ pc[SenderWriteID] = "sender_check_notify_data" /\ NotifyWrite
            \/ pc[SenderWriteID] = "sender_notify_data") /\ ReadLimit < Len(Got) + Len(Buffer)
        \* The sender is about to notify us of shutdown:
        \/ pc[SenderCloseID] \in {"sender_notify_closed"}
  \* If the writer is stuck, but there is now space available, the receiver will unblock it:
  /\ WriterShouldBeUnblocked
     => \* The reader is going to read more:
        \/ ReadLimit > Len(Got) /\ ReceiverLive
        \* The reader is about to increase WriteLimit:
        \/ (\/ pc[ReceiverReadID] = "recv_check_notify_read" /\ NotifyRead
            \/ pc[ReceiverReadID] = "recv_notify_read") /\ WriteLimit < Len(Got) + BufferSize
        \* The receiver is about to notify us of shutdown:
        \/ pc[ReceiverCloseID] \in {"recv_notify_closed"}
  /\ NotifyFlagsCorrect

If a process is on a path to being blocked then it must have set its notify flag. NotifyFlagsCorrect says that in that case, the flag it still set, or the interrupt has been sent, or the other process is just about to trigger the interrupt.

I managed to use that to prove that the sender’s steps preserved I, but I needed a little extra to finish the receiver proof. At this point, I finally spotted the obvious invariant (which you, no doubt, saw all along): whenever NotifyRead is still set, the sender has accurate information about the buffer.

1
2
3
4
5
/\ NotifyRead =>
      \* The sender has accurate information about the buffer:
      \/ WriteLimit = Len(Got) + BufferSize
      \* Or the flag is being cleared right now:
      \/ pc[ReceiverReadID] = "recv_check_notify_read"

That’s pretty obvious, isn’t it? The sender checks the buffer after setting the flag, so it must have accurate information at that point. The receiver clears the flag after reading from the buffer (which invalidates the sender’s information).

Now I had a dilemma. There was obviously going to be a matching property about NotifyWrite. Should I add that, or continue with just this? I was nearly done, so I continued and finished off the proofs.

With I proved, I was able to prove some other nice things quite easily:

1
2
3
4
5
THEOREM
  /\ I /\ SenderLive /\ ReceiverLive
  /\ \/ pc[SenderWriteID] = "sender_ready"
     \/ pc[SenderWriteID] = "sender_blocked" /\ ~SpaceAvailableInt
  => ReadLimit = Len(Got) + Len(Buffer)

That says that, whenever the sender is idle or blocked, the receiver will read everything sent so far, without any further help from the sender. And:

1
2
3
4
THEOREM
  /\ I /\ SenderLive /\ ReceiverLive
  /\ pc[ReceiverReadID] \in {"recv_await_data"} /\ ~DataReadyInt
  => WriteLimit = Len(Got) + BufferSize

That says that whenever the receiver is blocked, the sender can fill the buffer. That’s pretty nice. It would be possible to make a vchan system that e.g. could only send 1 byte at a time and still prove it couldn’t deadlock and would always deliver data, but here we have shown that the algorithm can use the whole buffer. At least, that’s what these theorems say as long as you believe that ReadLimit and WriteLimit are defined correctly.

With the proof complete, I then went back and deleted all the stuff about ReadLimit and WriteLimit from I and started again with just the new rules about NotifyRead and NotifyWrite. Instead of using WriteLimit = Len(Got) + BufferSize to indicate that the sender has accurate information, I made a new SenderInfoAccurate that just returns TRUE whenever the sender will fill the buffer without further help. That avoids some unnecessary arithmetic, which TLAPS needs a lot of help with.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
(* The sender's information is accurate if whenever it is going to block, the buffer
   really is full. *)
SenderInfoAccurate ==
  \* We have accurate information:
  \/ Len(Buffer) + free = BufferSize
  \* In these states, we're going to check the buffer before blocking:
  \/ pc[SenderWriteID] \in {"sender_ready", "sender_request_notify", "sender_write",
                            "sender_recheck_len", "sender_check_recv_live", "Done"}
  \/ pc[SenderWriteID] \in {"sender_request_notify"} /\ free < Len(msg)
  \* If we've been signalled, we'll immediately wake next time we try to block:
  \/ SpaceAvailableInt
  \* We're about to write some data:
  \/ /\ pc[SenderWriteID] \in {"sender_write_data"}
     /\ free >= Len(msg)                \* But we won't need to block
  \* If we wrote all the data we intended to, we'll return without blocking:
  \/ /\ pc[SenderWriteID] \in {"sender_check_notify_data", "sender_notify_data"}
     /\ Len(msg) = 0

By talking about accuracy instead of the write limit, I was also able to include “Done” in with the other happy cases. Before, that had to be treated as a possible problem because the sender can’t use the full buffer when it’s Done.

With this change, the proof of Spec => []I became much simpler (384 lines shorter). And most of the remaining steps were trivial.

The ReadLimit and WriteLimit idea still seemed useful, though, but I found I was able to prove the same things from I. e.g. we can still conclude this, even if I doesn’t mention WriteLimit:

1
2
3
4
THEOREM
  /\ I /\ SenderLive /\ ReceiverLive
  /\ pc[ReceiverReadID] \in {"recv_await_data"} /\ ~DataReadyInt
  => WriteLimit = Len(Got) + BufferSize

That’s nice, because it keeps the invariant and its proofs simple, but we still get the same result in the end.

I initially defined WriteLimit to be the number of bytes the sender could write if the sending application wanted to send enough data, but I later changed it to be the actual number of bytes it would write if the application didn’t try to send any more. This is because otherwise, with packet-based sends (where we only write when the buffer has enough space for the whole message at once) WriteLimit could go down. e.g. we think we can write another 3 bytes, but then the application decides to write 10 bytes and now we can’t write anything more.

The limit theorems above are useful properties, but it would be good to have more confidence that ReadLimit and WriteLimit are correct. I was able to prove some useful lemmas here.

First, ReceiverRead steps don’t change ReadLimit (as long as the receiver hasn’t closed the connection):

1
2
3
THEOREM ReceiverReadPreservesReadLimit ==
  ASSUME I, ReceiverLive, ReceiverRead
  PROVE  UNCHANGED ReadLimit

This gives us a good reason to think that ReadLimit is correct:

  • When the receiver is blocked it cannot read any more than it has without help.
  • ReadLimit is defined to be Len(Got) then, so ReadLimit is obviously correct for this case.
  • Since read steps preserve ReadLimit, this shows that ReadLimit is correct in all cases.

e.g. if ReadLimit = 5 and no other processes do anything, then we will end up in a state with the receiver blocked, and ReadLimit = Len(Got) = 5 and so we really did read a total of 5 bytes.

I was also able to prove that it never decreases (unless the receiver closes the connection):

1
2
3
THEOREM ReadLimitMonotonic ==
  ASSUME I, Next, ReceiverLive
  PROVE  ReadLimit' >= ReadLimit

So, if ReadLimit = n then it will always be at least n, and if the receiver ever blocks then it will have read at least n bytes.

I was able to prove similar properties about WriteLimit. So, I feel reasonably confident that these limit predictions are correct.

Disappointingly, we can’t actually prove Availability using TLAPS, because currently it understands very little temporal logic (see TLAPS limitations). However, I could show that the system can’t deadlock while there’s data to be transmitted:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
(* We can't get into a state where the sender and receiver are both blocked
   and there is no wakeup pending: *)
THEOREM DeadlockFree1 ==
  ASSUME I
  PROVE  ~ /\ pc[SenderWriteID] = "sender_blocked"
           /\ ~SpaceAvailableInt /\ SenderLive
           /\ pc[ReceiverReadID] = "recv_await_data"
           /\ ~DataReadyInt /\ ReceiverLive
<1> SUFFICES ASSUME /\ pc[SenderWriteID] = "sender_blocked"
                    /\ ~SpaceAvailableInt /\ SenderLive
                    /\ pc[ReceiverReadID] = "recv_await_data"
                    /\ ~DataReadyInt /\ ReceiverLive
             PROVE  FALSE
    OBVIOUS
<1> NotifyFlagsCorrect BY DEF I
<1> NotifyRead BY DEF NotifyFlagsCorrect
<1> NotifyWrite
    <2> have = 0 BY DEF IntegrityI, I
    <2> QED BY DEF NotifyFlagsCorrect
<1> SenderInfoAccurate /\ ReaderInfoAccurate BY DEF I
<1> free = 0 BY DEF IntegrityI, I
<1> Len(Buffer) = BufferSize BY DEF SenderInfoAccurate
<1> Len(Buffer) = 0 BY DEF ReaderInfoAccurate
<1> QED BY BufferSizeType

(* We can't get into a state where the sender is idle and the receiver is blocked
   unless the buffer is empty (all data sent has been consumed): *)
THEOREM DeadlockFree2 ==
  ASSUME I, pc[SenderWriteID] = "sender_ready", SenderLive,
         pc[ReceiverReadID] = "recv_await_data", ReceiverLive,
         ~DataReadyInt
  PROVE  Len(Buffer) = 0

I’ve included the proof of DeadlockFree1 above:

  • To show deadlock can’t happen, it suffices to assume it has happened and show a contradiction.
  • If both processes are blocked then NotifyRead and NotifyWrite must both be set (because processes don’t block without setting them, and if they’d been unset then an interrupt would now be pending and we wouldn’t be blocked).
  • Since NotifyRead is still set, the sender is correct in thinking that the buffer is still full.
  • Since NotifyWrite is still set, the receiver is correct in thinking that the buffer is still empty.
  • That would be a contradiction, since BufferSize isn’t zero.

If it doesn’t deadlock, then some process must keep getting woken up by interrupts, which means that interrupts keep being sent. We only send interrupts after making progress (writing to the buffer or reading from it), so we must keep making progress. We’ll have to content ourselves with that argument.

Experiences with TLAPS

The toolbox doesn’t come with the proof system, so you need to install it separately. The instructions are out-of-date and have a lot of broken links. In May, I turned the steps into a Dockerfile, which got it partly installed, and asked on the TLA group for help, but no-one else seemed to know how to install it either. By looking at the error messages and searching the web for programs with the same names, I finally managed to get it working in December. If you have trouble installing it too, try using my Docker image.

Once installed, you can write a proof in the toolbox and then press Ctrl-G, Ctrl-G to check it. On success, the proof turns green. On failure, the failing step turns red. You can also do the Ctrl-G, Ctrl-G combination on a single step to check just that step. That’s useful, because it’s pretty slow. It takes more than 10 minutes to check the complete specification.

TLA proofs are done in the mathematical style, which is to write a set of propositions and vaguely suggest that thinking about these will lead you to the proof. This is good for building intuition, but bad for reproducibility. A mathematical proof is considered correct if the reader is convinced by it, which depends on the reader. In this case, the “reader” is a collection of automated theorem-provers with various timeouts. This means that whether a proof is correct or not depends on how fast your computer is, how many programs are currently running, etc. A proof might pass one day and fail the next. Some proof steps consistently pass when you try them individually, but consistently fail when checked as part of the whole proof. If a step fails, you need to break it down into smaller steps.

Sometimes the proof system is very clever, and immediately solves complex steps. For example, here is the proof that the SenderClose process (which represents the sender closing the channel), preserves the invariant I:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
LEMMA SenderClosePreservesI ==
  I /\ SenderClose => I'
<1> SUFFICES ASSUME I, SenderClose
             PROVE  I'
    OBVIOUS
<1> IntegrityI BY DEF I
<1> TypeOK BY DEF IntegrityI
<1> PCOK BY DEF IntegrityI
<1>1. CASE sender_open
      <2> USE <1>1 DEF sender_open
      <2> UNCHANGED << pc[SenderWriteID], pc[ReceiverReadID], pc[ReceiverCloseID] >> BY DEF PCOK
      <2> pc'[SenderCloseID] = "sender_notify_closed" BY DEF PCOK
      <2> TypeOK' BY DEF TypeOK
      <2> PCOK' BY DEF PCOK
      <2> IntegrityI' BY DEF IntegrityI
      <2> NotifyFlagsCorrect' BY DEF NotifyFlagsCorrect, I
      <2> QED BY DEF I, SenderInfoAccurate, ReaderInfoAccurate, CloseOK
<1>2. CASE sender_notify_closed
      <2> USE <1>2 DEF sender_notify_closed
      <2> UNCHANGED << pc[SenderWriteID], pc[ReceiverReadID], pc[ReceiverCloseID] >> BY DEF PCOK
      <2> pc'[SenderCloseID] = "Done" BY DEF PCOK
      <2> TypeOK' BY DEF TypeOK
      <2> PCOK' BY DEF PCOK
      <2> IntegrityI' BY DEF IntegrityI
      <2> NotifyFlagsCorrect' BY DEF NotifyFlagsCorrect, I
      <2> QED BY DEF I, SenderInfoAccurate, ReaderInfoAccurate, CloseOK
<1>3. QED
  BY <1>1, <1>2 DEF SenderClose

A step such as IntegrityI' BY DEF IntegrityI says “You can see that IntegrityI will be true in the next step just by looking at its definition”. So this whole lemma is really just saying “it’s obvious”. And TLAPS agrees.

At other times, TLAPS can be maddeningly stupid. And it can’t tell you what the problem is - it can only make things go red.

For example, this fails:

1
2
3
4
5
THEOREM
  ASSUME pc' = [pc EXCEPT ![1] = "l2"],
         pc[2] = "l1"
  PROVE  pc'[2] = "l1"
OBVIOUS

We’re trying to say that pc[2] is unchanged, given that pc' is the same as pc except that we changed pc[1]. The problem is that TLA is an untyped language. Even though we know we did a mapping update to pc, that isn’t enough (apparently) to conclude that pc is in fact a mapping. To fix it, you need:

1
2
3
4
5
6
THEOREM
  ASSUME pc \in [Nat -> STRING],
         pc' = [pc EXCEPT ![1] = "l2"],
         pc[2] = "l1"
  PROVE  pc'[2] = "l1"
OBVIOUS

The extra pc \in [Nat -> STRING] tells TLA the type of the pc variable. I found missing type information to be the biggest problem when doing proofs, because you just automatically assume that the computer will know the types of things. Another example:

1
2
3
4
5
THEOREM
  ASSUME NEW x \in Nat, NEW y \in Nat,
         x + Min(y, 10) = x + y
  PROVE  Min(y, 10) = y
OBVIOUS

We’re just trying to remove the x + ... from both sides of the equation. The problem is, TLA doesn’t know that Min(y, 10) is a number, so it doesn’t know whether the normal laws of addition apply in this case. It can’t tell you that, though - it can only go red. Here’s the solution:

1
2
3
4
5
THEOREM
  ASSUME NEW x \in Nat, NEW y \in Nat,
         x + Min(y, 10) = x + y
  PROVE  Min(y, 10) = y
BY DEF Min

The BY DEF Min tells TLAPS to share the definition of Min with the solvers. Then they can see that Min(y, 10) must be a natural number too and everything works.

Another annoyance is that sometimes it can’t find the right lemma to use, even when you tell it exactly what it needs. Here’s an extreme case:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
LEMMA TransferFacts ==
  ASSUME NEW src, NEW src2,   \* (TLAPS doesn't cope with "NEW VARAIBLE src")
         NEW dst, NEW dst2,
         NEW i \in 1..Len(src),
         src \in MESSAGE,
         dst \in MESSAGE,
         dst2 = dst \o Take(src, i),
         src2 = Drop(src, i)
 PROVE  /\ src2 \in MESSAGE
        /\ Len(src2) = Len(src) - i
        /\ dst2 \in MESSAGE
        /\ Len(dst2) = Len(dst) + i
        /\ UNCHANGED (dst \o src)
PROOF OMITTED

LEMMA SameAgain ==
  ASSUME NEW src, NEW src2,   \* (TLAPS doesn't cope with "NEW VARAIBLE src")
         NEW dst, NEW dst2,
         NEW i \in 1..Len(src),
         src \in MESSAGE,
         dst \in MESSAGE,
         dst2 = dst \o Take(src, i),
         src2 = Drop(src, i)
 PROVE  /\ src2 \in MESSAGE
        /\ Len(src2) = Len(src) - i
        /\ dst2 \in MESSAGE
        /\ Len(dst2) = Len(dst) + i
        /\ UNCHANGED (dst \o src)
BY TransferFacts

TransferFacts states some useful facts about transferring data between two variables. You can prove that quite easily. SameAgain is identical in every way, and just refers to TransferFacts for the proof. But even with only one lemma to consider - one that matches all the assumptions and conclusions perfectly - none of the solvers could figure this one out!

My eventual solution was to name the bundle of results. This works:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
TransferResults(src, src2, dst, dst2, i) ==
  /\ src2 \in MESSAGE
  /\ Len(src2) = Len(src) - i
  /\ dst2 \in MESSAGE
  /\ Len(dst2) = Len(dst) + i
  /\ UNCHANGED (dst \o src)

LEMMA TransferFacts ==
  ASSUME NEW src, NEW src2,
         NEW dst, NEW dst2,
         NEW i \in 1..Len(src),
         src \in MESSAGE,
         dst \in MESSAGE,
         dst2 = dst \o Take(src, i),
         src2 = Drop(src, i)
 PROVE   TransferResults(src, src2, dst, dst2, i)
PROOF OMITTED

LEMMA SameAgain ==
  ASSUME NEW src, NEW src2,
         NEW dst, NEW dst2,
         NEW i \in 1..Len(src),
         src \in MESSAGE,
         dst \in MESSAGE,
         dst2 = dst \o Take(src, i),
         src2 = Drop(src, i)
 PROVE   TransferResults(src, src2, dst, dst2, i)
BY TransferFacts

Most of the art of using TLAPS is in controlling how much information to share with the provers. Too little (such as failing to provide the definition of Min) and they don’t have enough information to find the proof. Too much (such as providing the definition of TransferResults) and they get overwhelmed and fail to find the proof.

It’s all a bit frustrating, but it does work, and being machine checked does give you some confidence that your proofs are actually correct.

Another, perhaps more important, benefit of machine checked proofs is that when you decide to change something in the specification you can just ask it to re-check everything. Go and have a cup of tea, and when you come back it will have highlighted in red any steps that need to be updated. I made a lot of changes, and this worked very well.

The TLAPS philosophy is that

If you are concerned with an algorithm or system, you should not be spending your time proving basic mathematical facts. Instead, you should assert the mathematical theorems you need as assumptions or theorems.

So even if you can’t find a formal proof of every step, you can still use TLAPS to break it down into steps than you either can prove, or that you think are obvious enough that they don’t require a proof. However, I was able to prove everything I needed for the vchan specification within TLAPS.

The final specification

I did a little bit of tidying up at the end. In particular, I removed the want variable from the specification. I didn’t like it because it doesn’t correspond to anything in the OCaml implementation, and the only place the algorithm uses it is to decide whether to set NotifyWrite, which I thought might be wrong anyway.

I changed this:

1
2
recv_got_len:           if (have >= want) goto recv_read_data
                        else NotifyWrite := TRUE;

to:

1
2
3
4
5
6
recv_got_len:           either {
                          if (have > 0) goto recv_read_data
                          else NotifyWrite := TRUE;
                        } or {
                          NotifyWrite := TRUE;
                        };

That always allows an implementation to set NotifyWrite if it wants to, or to skip that step just as long as have > 0. That covers the current C behaviour, my proposed C behaviour, and the OCaml implementation. It also simplifies the invariant, and even made the proofs shorter!

I put the final specification online at spec-vchan. I also configured Travis CI to check all the models and verify all the proofs. That’s useful because sometimes I’m too impatient to recheck everything on my laptop before pushing updates.

You can generate a PDF version of the specification with make pdfs. Expressions there can be a little easier to read because they use proper symbols, but it also breaks things up into pages, which is highly annoying. It would be nice if it could omit the proofs too, as they’re really only useful if you’re trying to edit them. I’d rather just see the statement of each theorem.

The original bug

With my new understanding of vchan, I couldn’t see anything obvious wrong with the C code (at least, as long as you keep the connection open, which the firewall does).

I then took a look at ocaml-vchan. The first thing I noticed was that someone had commented out all the memory barriers, noting in the Git log that they weren’t needed on x86. I am using x86, so that’s not it, but I filed a bug about it anyway: Missing memory barriers.

The other strange thing I saw was the behaviour of the read function. It claims to implement the Mirage FLOW interface, which says that read “blocks until some data is available and returns a fresh buffer containing it”. However, looking at the code, what it actually does is to return a pointer directly into the shared buffer. It then delays updating the consumer counter until the next call to read. That’s rather dangerous, and I filed another bug about that: Read has very surprising behaviour. However, when I checked the mirage-qubes code, it just takes this buffer and makes a copy of it immediately. So that’s not the bug either.

Also, the original bug report mentioned a 10 second timeout, and neither the C implementation nor the OCaml one had any timeouts. Time to look at QubesDB itself.

QubesDB accepts messages from either the guest VM (the firewall) or from local clients connected over Unix domain sockets. The basic structure is:

1
2
3
4
5
6
while True:
  await vchan event, local client data, or 10 second timeout
  while vchan.receive_buffer non-empty:
    handle_vchan_data()
  for each ready client:
    handle_client_data()

The suspicion was that we were missing a vchan event, but then it was discovering that there was data in the buffer anyway due to the timeout. Looking at the code, it does seem to me that there is a possible race condition here:

  1. A local client asks to send some data.
  2. handle_client_data sends the data to the firewall using a blocking write.
  3. The firewall sends a message to QubesDB at the same time and signals an event because the firewall-to-db buffer has data.
  4. QubesDB gets the event but ignores it because it’s doing a blocking write and there’s still no space in the db-to-firewall direction.
  5. The firewall updates its consumer counter and signals another event, because the buffer now has space.
  6. The blocking write completes and QubesDB returns to the main loop.
  7. QubesDB goes to sleep for 10 seconds, without checking the buffer.

I don’t think this is the cause of the bug though, because the only messages the firewall might be sending here are QDB_RESP_OK messages, and QubesDB just discards such messages.

I managed to reproduce the problem myself, and saw that in fact QubesDB doesn’t make any progress due to the 10 second timeout. It just tries to go back to sleep for another 10 seconds and then immediately gets woken up by a message from a local client. So, it looks like QubesDB is only sending updates every 10 seconds because its client, qubesd, is only asking it to send updates every 10 seconds! And looking at the qubesd logs, I saw stacktraces about libvirt failing to attach network devices, so I read the Xen network device attachment specification to check that the firewall implemented that correctly.

I’m kidding, of course. There isn’t any such specification. But maybe this blog post will inspire someone to write one…

Conclusions

As users of open source software, we’re encouraged to look at the source code and check that it’s correct ourselves. But that’s pretty difficult without a specification saying what things are supposed to do. Often I deal with this by learning just enough to fix whatever bug I’m working on, but this time I decided to try making a proper specification instead. Making the TLA specification took rather a long time, but it was quite pleasant. Hopefully the next person who needs to know about vchan will appreciate it.

A TLA specification generally defines two sets of behaviours. The first is the set of desirable behaviours (e.g. those where the data is delivered correctly). This definition should clearly explain what users can expect from the system. The second defines the behaviours of a particular algorithm. This definition should make it easy to see how to implement the algorithm. The TLC model checker can check that the algorithm’s behaviours are all acceptable, at least within some defined limits.

Writing a specification using the TLA notation forces us to be precise about what we mean. For example, in a prose specification we might say “data sent will eventually arrive”, but in an executable TLA specification we’re forced to clarify what happens if the connection is closed. I would have expected that if a sender writes some data and then closes the connection then the data would still arrive, but the C implementation of vchan does not always ensure that. The TLC model checker can find a counter-example showing how this can fail in under a minute.

To explain why the algorithm always works, we need to find an inductive invariant. The TLC model checker can help with this, by presenting examples of unreachable states that satisfy the invariant but don’t preserve it after taking a step. We must add constraints to explain why these states are invalid. This was easy for the Integrity invariant, which explains why we never receive incorrect data, but I found it much harder to prove that the system cannot deadlock. I suspect that the original designer of a system would find this step easy, as presumably they already know why it works.

Once we have found an inductive invariant, we can write a formal machine-checked proof that the invariant is always true. Although TLAPS doesn’t allow us to prove liveness properties directly, I was able to prove various interesting things about the algorithm: it doesn’t deadlock; when the sender is blocked, the receiver can read everything that has been sent; and when the receiver is blocked, the sender can fill the entire buffer.

Writing formal proofs is a little tedious, largely because TLA is an untyped language. However, there is nothing particularly difficult about it, once you know how to work around various limitations of the proof checkers.

You might imagine that TLA would only work on very small programs like libvchan, but this is not the case. It’s just a matter of deciding what to specify in detail. For example, in this specification I didn’t give any details about how ring buffers work, but instead used a single Buffer variable to represent them. For a specification of a larger system using vchan, I would model each channel using just Sent and Got and an action that transferred some of the difference on each step.

The TLA Toolbox has some rough edges. The ones I found most troublesome were: the keyboard shortcuts frequently stop working; when a temporal property is violated, it doesn’t tell you which one it was; and the model explorer tooltips appear right under the mouse pointer, preventing you from scrolling with the mouse wheel. It also likes to check its “news feed” on a regular basis. It can’t seem to do this at the same time as other operations, and if you’re in the middle of a particularly complex proof checking operation, it will sometimes suddenly pop up a box suggesting that you cancel your job, so that it can get back to reading the news.

However, it is improving. In the latest versions, when you get a syntax error, it now tells you where in the file the error is. And pressing Delete or Backspace while editing no longer causes it to crash and lose all unsaved data. In general I feel that the TLA Toolbox is quite usable now. If I were designing a new protocol, I would certainly use TLA to help with the design.

TLA does not integrate with any language type systems, so even after you have a specification you still need to check manually that your code matches the spec. It would be nice if you could check this automatically, somehow.

One final problem is that whenever I write a TLA specification, I feel the need to explain first what TLA is. Hopefully it will become more popular and that problem will go away.

Update 2019-01-10: Marek Marczykowski-Górecki told me that the state model for network devices is the same as the one for block devices, which is documented in the blkif.h block device header file, and provided libvirt debugging help - so the bug is now fixed!


churchturing.org / 2019-01-20T10:47:00