Respiration metaphor

Written by  on April 17, 2014

Awhile back, I was caught off guard when a roomful of lecture attendees scoffed at my rejection of the idea that forests breathe. I hadn’t thought very deeply about it. But it seemed (still seems) like a sloppy metaphor rather than something anyone would seriously defend, which they did. But I found this definition useful:

Any of the various analogous processes by which there is an exchange of gases.

So, as an analogy, I can live with it. Sure, gasses are exchanged by a forest differently than, say, by a grassland or by the ocean. But in order to take the analogy seriously, i.e. use it to think, I’d need something more similar to a single organism than a forest. For example, an aspen colony or even a slime mold or biofilm. But a forest is just too much of a collective of organisms for me to really think of it as a single organism.

I mean no disrespect to the Big Thinkers out there who rely on analogical subsumption in order to think bigger thoughts. I do it myself all the time. But we need to be honest about our (purposeful) sloppiness when doing so.

petitio principii

Written by  on February 19, 2014

My SO is currently taking a course that uses this book as a text book. The argument to be analyzed for fallacies is:

The death penalty is wrong because it’s murder.

She answered by saying the argument includes a persuasive definition, but commits none of the fallacies asked for in this homework assignment. The listed fallacies included “begging the question/circular argument/petitio principii”. Her reasoning for her answer was that the argument is basically this:

  • Murder is wrong.
  • The death penalty is murder.
  • Hence, the death penalty is wrong.

Now, the professor wrote off her answer almost flippantly, without even considering it. I can understand why, since this argument bears a striking similarity to an argument that is presented almost ubiquitously as canonical begging the question. That canonical example is: “The death penalty is immoral because killing is immoral.” This fits the typical tautological structure of begging the question because the death penalty is most definitely a form of killing. So, the conclusion is simply a more specific statement of the premise.

But the argument we’re considering, here, is most definitetly not that simple. To someone not familiar with the technical definitions, Murder may be synonymous with killing. After all, murder is simply illegal killing. Hence, murder and the death penalty are specific types of killing. What makes the canonical argument petitio principii is that the generality of killing contains the specific death penalty. But that’s not what this argument is doing.

Rather, the flaw in this argument is simply an undefended premise: the death penalty is murder. We can’t infer whether the arguer intends the death penalty to be a sub-type of murder or synonymous with it. But it’s a substantive premise, not simply an obvious rephrasing of the conclusion.

Now, I have seen some definitions of begging the question that include implicit premises as a way to commit the fallacy. But I think that’s a serious miscategorization. Concealed premises that cause fallacy are premises that would not be acceptable to lots of people, i.e. obviously unsound or “tricky” premises (arguably as Aristotle points out).

I can only assume that if my SO had included additional argument for that second premise, the professor might have paid closer attention.

Why do I care? Because petitio principii holds a special place in modeling and simulation, identified as “inscription error” by Brian Cantewell Smith, or better termed “pre-emptive registration”. The idea is that when building a simulation from which you intend to predict or discover interesting behaviors, the behavior you observe is not very interesting if you purposefully programmed the simulation to behave that way. Now, there’s an interesting twist to this in that magic word “purposefully”. One can make the argument that accidental inscription is what simulation is all about! And this twist extends all the way out into the foundations of logic and mathematics, perhaps even physics and chemistry. The entire enterprise of science may well be about computing forward from premises to conclusions so as to see what “concealed premises” are hidden therein. Indeed, this is, I think, the fundamental criticism of computationalism. If the universe is simply deductive computation and all deduction is tantamount to tautology, then we’re all just wasting our time, here… really. ;-)

Saving Server Configurations Under Git

Written by  on January 16, 2014

At my current work we create a git repository per server and check in the relevant configuration files order to keep track of work done on individual servers.

  • for each server you have to make a repo on git.
    git:/home/pdxadmin $ cd serverconfigs
    git:/home/pdxadmin/serverconfigs $ mkdir smtp-etc.git
    git:/home/pdxadmin/serverconfigs $ cd smtp-etc.git
    git:/home/pdxadmin/serverconfigs/smtp-etc.git $ git init --bare
    Initialized empty Git repository in /home/pdxadmin/serverconfigs/smtp-etc.git/
    git:/home/pdxadmin/serverconfigs/smtp-etc.git $ git update-server-info
    git:/home/pdxadmin/serverconfigs/smtp-etc.git $ echo "postfix">description
    git:/home/pdxadmin/serverconfigs $ cd ..
  • create a dsa key and send it to git.
    smtp:/etc # ssh-keygen -t dsa
    Generating public/private dsa key pair.
    Enter file in which to save the key (/root/.ssh/id_dsa): 
    Enter passphrase (empty for no passphrase): 
    Enter same passphrase again: 
    Your identification has been saved in /root/.ssh/id_dsa.
    Your public key has been saved in /root/.ssh/id_dsa.pub.
    The key fingerprint is:
    4d:bd:a0:0c:1a:e8:46:64:6a:47:fc:cd:ed:f4:5e:fe root@smtp.NGI-dev
    The key's randomart image is:
    +--[ DSA 1024]----+
    |  +.             |
    | +.o       .     |
    |..o.o + . o .    |
    |.o.  + = * . .   |
    |  o .   S o .    |
    | .       . . .   |
    |          . o    |
    |           . .   |
    |              .E |
    +-----------------+
    smtp:/etc # scp ~/.ssh/id_dsa.pub pdxadmin@git:/tmp/
    The authenticity of host 'git (192.168.0.103)' can't be established.
    ECDSA key fingerprint is 50:0c:88:f3:8f:13:e0:7c:79:0a:b9:f8:d5:90:dc:5b.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'git,192.168.0.103' (ECDSA) to the list of known hosts.
    Password: 
    id_dsa.pub                                                                                                         100%  607     0.6KB/s   00:00    
    smtp:/etc # git:/home/pdxadmin/serverconfigs $
  • add it to authorized keys
    git:/home/pdxadmin $ cat /tmp/id_dsa.pub >>~/.ssh/authorized_keys
  • create local repo and push it to git.
    smtp:/etc # zypper install git-core
    smtp:/etc # git init
    Initialized empty Git repository in /etc/.git/
    smtp:/etc # git add postfix
    smtp:/etc # git commit -m "initial commit / postfix directory"
    [master (root-commit) 6f827ba] initial commit / postfix directory
     Committer: root <root@smtp.NGI-dev>
    ...
     create mode 100644 postfix/virtual.db
    smtp:/etc # git remote add origin pdxadmin@git:serverconfigs/smtp-etc.git
    smtp:/etc # git push -u origin master
    Counting objects: 62, done.
    Compressing objects: 100% (60/60), done.
    Writing objects: 100% (62/62), 68.57 KiB, done.
    Total 62 (delta 27), reused 0 (delta 0)
    To pdxadmin@git:serverconfigs/smtp-etc.git
     * [new branch]      master -> master
    Branch master set up to track remote branch master from origin.
    smtp:/etc #

The Closing of the Scientific Mind

Written by  on January 8, 2014

The Closing of the Scientific Mind

I find a lot to like about this article. But there are 2 irritants:

  1. sore misrepresentation of computationalism, and
  2. implicit accusation of false analogy surrounded by prose rife with false analogy.

(1) Computationalism does not blindly assert that a phone app could be conscious. Acceptance of the Church-Turing thesis is common. But there are plenty of scientists (even computer science people if you’re one to call them “scientists”) recognize that the structure of the machine matters not only to what type of programs will run (at all, or well) on that machine, but also to whether any given software and program state can be moved from one to another.

Anyone who believes in the caricature of computationalism as Gelernter describes it cannot actually work with computers, especially not embedded computers.

(2) “Man is only a computer if you ignore everything that distinguishes him from a computer.” Followed immediately by: “science today is the Catholic Church around the start of the 16th century”. I realize I’m committing tu quoque … but sheesh.

Backing up your github content.

Written by  on December 16, 2013

I never like trusting others to care for my things as well as I do. Github, bitbucket, and googledocs are big piles of trust. So here is a quicky to copy and pull from all of your users repositories.


#!/usr/bin/python
# Quick and dirty backup of your git files.
from pygithub3 import Github
import git
import os
import subprocess
os.chdir('/home/git/repositories')
gh = Github(login='git')
for user in ['JoeDumoulin','soycamo','suspect-devices','feurig'] :
    for repo in gh.repos.list(user).all():
        try:
            os.stat(repo.full_name.split('/')[0])
        except:
            os.mkdir(repo.full_name.split('/')[0])       

        print repo.git_url,'->', repo.full_name.split('/')[0]
        try:
            git.Repo.clone_from(repo.git_url, repo.full_name)
            description_file=open(repo.full_name+"/.git/description","r+")
            if("Unnamed repository" in description_file.read()):
               description_file.seek(0)
               print repo.description
               description_file.write(repo.description)
               description_file.truncate()
               description_file.close()
        except Exception as e:
            if str(e).find('already exists and is not an empty'):
               try:
                  git.cmd.Git(repo.full_name).pull()
                  print repo.full_name, "should be up to date"
               except Exception as e2:
                  print str(e2)
                  pass
            else: 
               print str(e)
            pass

subprocess.call(["chown", "-R","git:git","."]) 

Notes:

The two python tools we use are pygithub3 http://pygithub3.readthedocs.org/en/latest/index.html which gives us access to githubs api. To work with the git repositories we use GitPython http://pythonhosted.org/GitPython/0.3.2/index.html.

We have been looking at gitweb which uses the description file in the .git repository. Since many of these repos are created on github the description file is empty though github provides the description in its json, so we populate it.

CryptoLocker

Written by  on November 27, 2013

Renee’ clicked on a CryptoLocker attachment. [sigh] I’ll be damned if I’m going to reward those defectors for their behavior. So, we went to a local computer shop and bought a new one for her. (That allows me to study the infected one at my leisure.) She lost quite a lot of work. But nothing time sensitive that she can’t regenerate over time.

Normally, I’m relatively accepting of black-hat behavior. Cracking into low-hanging fruit and demonstrating flaws in the systems of the sanctimonious, although still defector behavior, serve a useful co-evolutionary purpose.

But I can’t abide hostage taking. And to anyone tempted to pay the ransom: Please don’t reward them. The more money they make, the more depraved they’ll be.

Gnome3 lockscreen background

Written by  on November 14, 2013

For whatever reason, sometimes when I lock my screen it eventually blanks and sometimes it doesn’t. Since I prefer solid black backgrounds on pretty much everything anyway, I figured I’d change the lockscreen background and GDM login screen to my usual black-pixel background.

Here’s how I did it:

$ cd /usr/share/images/desktop-base
$ cp joy.xml black.xml

edit black.xml like so:

5,8c5
< /usr/share/images/desktop-base/joy-wallpaper_1280x1024.svg
< /usr/share/images/desktop-base/joy-wallpaper_1600x1200.svg
< /usr/share/images/desktop-base/joy-wallpaper_1920x1080.svg
< /usr/share/images/desktop-base/joy-wallpaper_1920x1200.svg
---
> /usr/share/images/desktop-base/black-pixel.png

then

$ sudo update-alternatives --install /usr/share/images/desktop-background.xml desktop-background.xml /usr/share/images/desktop-base/black.xml 100

Then logout and login, check your lockscreen.

Presenting equations

Written by  on August 8, 2013

It took me way too long to come up with a scalable representation of a tiny bit of math.  In order to log it for posterity, I figured, I’d describe it briefly here.  What worked was to:

  1. Generate the equation in LyX.
  2. Export it into HTML.
  3. Make it frickin’ huge in Firefox.
  4. Take a screenshot and crop it.
  5. Trace the bitmap in Inkscape using brightness cutoff 0.90.
  6. Save the path as a plain SVG.
  7. Import it into LibreOffice.

Just to be complete, here’s the equation:

Cout(t) =  − 1MQexp1 − 1 + 4DNT(s + g(s))2DN,
where g(s) = k1 − k1k2s + ke + k2 + a − abs + k1 − k1k2s + ke + k2 + b

 

WIREs article

Written by  on May 16, 2013

Yay!  Our latest article is finally out:

Agent-based modeling: a systematic assessment of use cases and requirements for enhancing pharmaceutical research and development productivity

Here’s the description of it on our ABM website.

New paper: Agent-based modeling: a systematic assessment of use cases and requirements for enhancing pharmaceutical research and development productivity, published in Wiley Interdisciplinary Reviews: Systems Biology and Medicine. This one walks through various use cases extracted and derived from various more tightly focused modeling and simulation papers. It also lays out a vision for an analog based model repository, including the features such a repository requires in order to satisfy its use cases. In particular, we talk about the need for embedded knowledge inside models.

 

Types of Pluralism

Written by  on May 15, 2013

This paper:

Pluralism, entwinement, and the levels of selection by Robert A. Wilson

distinguishes between object pluralism (consideration of multiple things, in this case the unit of evolutionary selection) and model pluralism (consideration of the putatively the same thing with multiple models). The FURM that I’ve been working on for quite some time, relies on the latter, model pluralism.  More specifically, it is a parallax method intended to steadily pare down the space of possibly accurate models to a manageable size.

What Wilson’s article brought to the foreground for me, however, is the ontological commitment to the existence of the objects (or their attributes), in this case the levels of selection. I seem to subscribe to an ur-pluralism, an agnosticism regarding the very existence of the objects being modeled (aka the referent). This is not an anti-realist position. The referent can exist. But it is an anti-{naive realist} position. The attributes of the referent that you perceive or infer may not exist, e.g. “levels”. I sporadically rant about the inadequacy of the term “level” with respect to modeling complex systems. But I’m always treated to confused expressions and (to me) weird reactions. And while I relish confusion and weirdness (Hail Eris! =><=), perhaps this will help me navigate it more effectively.

Wrapping it up. A little wsgi, a little soup, a little xlwt.

Written by  on February 25, 2013

I wanted to create an easy way for my manager to pull the monthly uptime reports from nagios so I wrapped the script I wrote last week into a webapp2 based report generator which takes on two types of requests

  • /reports/generator/monthly?month=[n]
  • /reports/generator/timespan?fromdate=Dow Mon dd yyyy&todate=Dow Mon dd yyyy

    The date range format was determined by the easiest to find and impliment javascript date picker I could find. It’s called pickaday http://dbushell.com/2012/10/09/pikaday-javascript-datepicker/

    Since WSGI was already required to run up the graphix server on our monitoring box I only needed to run up the webapp2 framework.It would be nice if there were a package for it but instead we had to install the packages from python using pip. http://webapp-improved.appspot.com/tutorials/quickstart.nogae.html

    $ pip install WebOb
    $ pip install Paste
    $ pip install webapp2
    

    One of the things I keep struggling with is how to mix and map wsgi apps with “normal” apache this is what I wound up with.

            WSGIScriptAlias /reports/generator /var/www/reports/generator/nag2xls.wsgi
    
            <Directory "/var/www/reports/generator">
                    options +ExecCGI
                    AddHandler wsgi-script .wsgi
                    Order deny,allow
                    Allow from all
            </Directory>
    

    Most of the resulting app is getting the date range localized and as a unix timestamp for nagios. The rest of it is more or less what I already written to create the spreadsheet.

    import cgi
    import webapp2
    from BeautifulSoup import BeautifulSoup
    import urllib2, base64
    import xlwt
    import datetime
    import time
    
    
    class downloadMonthlyReport(webapp2.RequestHandler):
        def get(self):
            today=datetime.datetime.utcnow()
            theMonth=int(self.request.get('month'))
            print "theMonth=%d" % theMonth
            fromDate=datetime.datetime(today.year,theMonth,1,0,0,0)+datetime.timedelta(seconds=time.timezone) 
            if (theMonth > 11):
                toDate=datetime.datetime(today.year+1,1,1,0,0,0)+datetime.timedelta(seconds=time.timezone-1)
            else:
                toDate=datetime.datetime(today.year,theMonth+1,1,0,0,0)+datetime.timedelta(seconds=time.timezone-1)
            #print fromDate, "->", toDate
            fromEpoch=int((fromDate - datetime.datetime(1970,1,1)).total_seconds()) 
            toEpoch=int((toDate - datetime.datetime(1970,1,1)).total_seconds())
            self.response.headers['Content-Type'] = 'application/vnd.ms-excel'
            self.response.headers['Content-Disposition'] = 'attachment;filename="Uptime%s.xls"' % fromDate.strftime("%b%y")
            
            return build_sheet(self.response.out,'xxxx','xxxx',fromEpoch,toEpoch)
    
            
    class downloadTimeFrameReport(webapp2.RequestHandler):
        def get(self):
            #print self.request
            aDayMinusASecond=86399
            fromDate=int(time.mktime(time.strptime(self.request.get('fromdate'),'%a %b %d %Y')))
            toDate=int(time.mktime(time.strptime(self.request.get('todate'),'%a %b %d %Y'))) + aDayMinusASecond
            #print fromDate, "->", toDate
            self.response.headers['Content-Type'] = 'application/vnd.ms-excel'
            self.response.headers['Content-Disposition'] = 'attachment;filename="Uptime%s-%s.xls"' % (
                                             datetime.datetime.fromtimestamp(fromDate).strftime("%d%b%y"),
                                             datetime.datetime.fromtimestamp(toDate).strftime("%d%b%y"),
                                            )
                                                             
            return build_sheet(self.response.out,'xxxxx','xxxxx',fromDate,toDate)
            
            
    application = webapp2.WSGIApplication([('/reports/generator/timeframe.*', downloadTimeFrameReport),
                                          # ('/reports/generator/monthly.*', downloadMonthlyReport)
                                           ('/.*', downloadMonthlyReport)
                                           ],
                                  debug=True)
    
    def build_sheet(myfile,username,password,startTime,endTime):
        
        reportRequest='http://idcctrl1/cgi-bin/nagios3/avail.cgi?t1=%d&t2=%d&show_log_entries=&hostgroup=all' % ( startTime , endTime)
        print reportRequest
        request = urllib2.Request(reportRequest)
        base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
        request.add_header("Authorization", "Basic %s" % base64string)   
        result = urllib2.urlopen(request).read()
        soup = BeautifulSoup(''.join(result))
        
        workbook = xlwt.Workbook()
        sheet1 = workbook.add_sheet('By Group')
        sheet2 = workbook.add_sheet('By Server')
        
        row_count = 0
        s2rc=0
        maxCol1Width=0
        
        littleFont = xlwt.Font() # Create the Font
        littleFont.name = 'Times New Roman'
        littleFont.height = 0x00A0 # ~8pt
        littleFont.italic = True
        lableStyle=xlwt.XFStyle()
        lableStyle.font = littleFont
        
        emFont = xlwt.Font() # Create the Font
        emFont.name = 'Times New Roman'
        #emFont.height = 0x00A0 # ~8pt
        emFont.italic = True
        emFont.bold = True
        headerStyle=xlwt.XFStyle()
        headerStyle.font = emFont
        
        percentStyle = xlwt.XFStyle()
        percentStyle.num_format_str = '##0.00%' 
        
        sheet1.write(row_count,0,
                     "Uptime Report : %s to %s" % (
                       datetime.datetime.fromtimestamp(startTime).strftime("%d/%b/%y"),
                       datetime.datetime.fromtimestamp(endTime).strftime("%d/%b/%y")
                       ),
                      headerStyle)
        row_count+=1
                       
        sheet1.write(row_count,0,'Host Name',lableStyle) 
        sheet1.write(row_count,1,'Up',lableStyle) 
        sheet1.write(row_count,2,'Down',lableStyle) 
        sheet1.write(row_count,3,'Unreachable',lableStyle) 
        sheet1.write(row_count,4,'Unknown',lableStyle)
        row_count+=1
         
        for header in soup.findAll("div", { "class" : "dataTitle" }):
            title = header.find(text=True)
            if "All Hostgroups" in title: 
                continue
            #print "\n",title
            row_count+=1
            sheet1.write(row_count,0,title,headerStyle)
            
            table = header.findNext('table', { "class" : "data" })
            try:  
                rows = table.findAll('tr')
                for tr in rows:
                    cols = tr.findAll('td')
                    column=0
                    skipThisRow=False
                    for td in cols:
                        text = ''.join(td.find(text=True)).split(' ',1)[0]
                        if column==0:
                            cw=len(text)*256
                            if cw>maxCol1Width:
                                maxCol1Width=cw
                                sheet1.col(0).width = maxCol1Width
                                sheet2.col(0).width = maxCol1Width
                            if 'Average' in text:
                                sheet1.write(row_count,column,text.strip('%'),headerStyle)
                            else:
                                sheet1.write(row_count,column,text.strip('%'))
                        else:
                            sheet1.write(row_count,column,float(text.strip('%'))/100.00,percentStyle)
                        if column==0:
                            if 'Average' in text:
                                skipThisRow=True
                        if not skipThisRow and column<2:
                            if column==0:
                                sheet2.write(s2rc,column,text.strip('%'))
                            else: 
                                sheet2.write(s2rc,column,float(text.strip('%'))/100.00,percentStyle)
                        column+=1
                        #print text+"|",
                    if (skipThisRow is False and column>0):
                        s2rc+=1
                    #print
                    row_count+=1
            except Exception as e:
                print str(e)
        
        workbook.save(myfile)
        
    
    

Pulling uptime reports (straight to excel) from Nagios.

Written by  on February 18, 2013

The other day I was looking at automating the clean-up of the CA uptime reports into documents that could be turned into graphs using powershell. But really the end goal here is to get rid of the CA suit since no one employed here knows how to use it, it takes 5 systems and costs an exorbitant amount of money each year.

Since we had had a few outages in the past few weeks I thought I would look at the uptime reports available from Nagios, they are as good as the CA in terms of usable content and based on the outages we had experienced they seemed pretty accurate.

What was missing was to get them into a csv or excell format. Most of the searching I did lead me to a lot of people who hacked the source code to produce csv instead of the html pictured above. I found an easier way using a couple of python libraries that I have been checking out.

BeautifulSoup — a parsing library for xml/html
xlwt — a library for creating excel files directly

The following code creates a two sheet spreadsheet one broken out by group and one with just the by server uptimes.

#!/usr/bin/python
#TODO make a webapp2 wrapper for this.
#
from BeautifulSoup import BeautifulSoup
import urllib2, base64
import xlwt
username='xxxxx'
password='xxxxx'
spreadsheet='../Desktop/xfer/uptime.xls'

startTime=1358556160
endTime=1361234560

reportRequest='http://idcctrl1/cgi-bin/nagios3/avail.cgi?t1=%d&t2=%d&show_log_entries=&hostgroup=all' % ( startTime , endTime )
request = urllib2.Request(reportRequest)
base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
result = urllib2.urlopen(request).read()
soup = BeautifulSoup(''.join(result))

workbook = xlwt.Workbook()
sheet1 = workbook.add_sheet('By Group')
sheet2 = workbook.add_sheet('By Server')

row_count = 0
s2rc=0
maxCol1Width=0

littleFont = xlwt.Font() # Create the Font
littleFont.name = 'Times New Roman'
littleFont.height = 0x00A0 # ~8pt
littleFont.italic = True
lableStyle=xlwt.XFStyle()
lableStyle.font = littleFont

emFont = xlwt.Font() # Create the Font
emFont.name = 'Times New Roman'
#emFont.height = 0x00A0 # ~8pt
emFont.italic = True
emFont.bold = True
headerStyle=xlwt.XFStyle()
headerStyle.font = emFont

percentStyle = xlwt.XFStyle()
percentStyle.num_format_str = '##0.00%' 

sheet1.write(row_count,0,'Host Name',lableStyle)
sheet1.write(row_count,1,'Up',lableStyle)
sheet1.write(row_count,2,'Down',lableStyle)
sheet1.write(row_count,3,'Unreachable',lableStyle)
sheet1.write(row_count,4,'Unknown',lableStyle)
row_count+=1

for header in soup.findAll("div", { "class" : "dataTitle" }):
    title = header.find(text=True)
    if "All Hostgroups" in title:
        continue
    print "\n",title
    row_count+=1
    sheet1.write(row_count,0,title,headerStyle)

    table = header.findNext('table', { "class" : "data" })
    try:
        rows = table.findAll('tr')
        for tr in rows:
            cols = tr.findAll('td')
            column=0
            skipThisRow=False
            for td in cols:
                text = ''.join(td.find(text=True)).split(' ',1)[0]
                if column==0:
                    cw=len(text)*256
                    if cw>maxCol1Width:
                        maxCol1Width=cw
                        sheet1.col(0).width = maxCol1Width
                        sheet2.col(0).width = maxCol1Width
                    if 'Average' in text:
                        sheet1.write(row_count,column,text.strip('%'),headerStyle)
                    else:
                        sheet1.write(row_count,column,text.strip('%'))
                else:
                    sheet1.write(row_count,column,float(text.strip('%'))/100.00,percentStyle)
                if column==0:
                    if 'Average' in text:
                        skipThisRow=True
                if not skipThisRow and column0):
                s2rc+=1
            print
            row_count+=1
    except Exception as e:
        print str(e)

workbook.save(spreadsheet)

 

Visualizing thrash.

Written by  on February 12, 2013

For the past few months there have been consultants helping our dba’s get the oracle financial servers to oracle 12. They have been bringing the servers (a few of which are under memoried) to their knees. So they asked us to monitor memory processor and swap during their upgrades. We have the very expensive and labour intensive CAsuit of monitoring but when I went to look at the performance graphs they showed a swap usage of 11380.87% utilization, and blank graphs.

After a bit of research (pertinent links below) I found a this thing called graphite. Graphite has 3 components: a daemon to capture datapoints (carbon), a way to store them effectively (whisper) and a presentation layer (graphite).

I followed mostly this blog http://coreygoldberg.blogspot.com/2012/04/installing-graphite-099-on-ubuntu-1204.html which worked on my fresh 12.04 desktop. It didn’t work so well on the monitoring system or on my 10.4 based server but when I got stuck I just installed from source http://graphite.readthedocs.org/en/0.9.10/install-source.html Oh AND 12.04 does not have ceres library which for some reason doesnt show up in the pre-reqs test git it at https://github.com/graphite-project/ceres .

 


I was quickly able to put up the information that we wanted to monitor but when I tried to work on one of the solaris zones I started seeing negative memory usage.

load averages:  1.60,  1.63,  1.54;               up 94+13:00:11       17:22:10
285 processes: 283 sleeping, 2 on cpu
CPU states: 97.0% idle,  1.5% user,  1.5% kernel,  0.0% iowait,  0.0% swap
Kernel: 3439 ctxsw, 121 trap, 3248 intr, 3998 syscall, 96 flt, 4296 pgin
Memory: 35G phys mem, 78G free mem, 35G total swap, 29G free swap

The zone has 35G of memory allocated to it. But top sees the memory for the entire system. (this is probably the same mistake that CA was making) So on machines which are not global zones we use vmstat to get the free memory.

#!/usr/bin/env python

import platform
import socket
import time
import subprocess

CARBON_SERVER = '192.168.116.7'
CARBON_PORT = 2003
DELAY = 15  # secs

totalmem=1
meminuse=1
totalswap=1
swapinuse=1

def MGKtoZeros(numstr):
    K=1024
    M=K*1024
    G=M*1024
    if (numstr.rfind('G')>0):
        return float(numstr.replace('G',''))*G
    if (numstr.rfind('M')>0):
        return float(numstr.replace('M',''))*M
    if (numstr.rfind('K')>0):
        return float(numstr.replace('K',''))*K

def get_memstats():
    p=subprocess.Popen("/opt/it/bin/top -b|grep -i memory",
                       shell=True, stdout=subprocess.PIPE)
    o=p.communicate()[0]
    p=subprocess.Popen("/opt/it/bin/top -b|grep -i states",
                       shell=True, stdout=subprocess.PIPE)
    c=p.communicate()[0]
    p=subprocess.Popen("vmstat|tail -1",
                       shell=True, stdout=subprocess.PIPE)
    v=p.communicate()[0]
    tm=MGKtoZeros(o.split()[1])
    #mu=float(v.split()[4])
    mu=MGKtoZeros(o.split()[4])
    ts=MGKtoZeros(o.split()[7])
    su=MGKtoZeros(o.split()[10])
    cpu=100.00-float(c.split()[2].strip('%'))
    #su=float(v.split()[3])
    return (tm,tm-mu,ts,ts-su,cpu)

def send_msg(message):
    print 'sending message:\n%s' % message
    sock = socket.socket()
    sock.connect((CARBON_SERVER, CARBON_PORT))
    sock.sendall(message)
    sock.close()

if __name__ == '__main__':
    node = platform.node().replace('.', '-')
    while True:
      try:
        timestamp = int(time.time())
        (totalmem,meminuse,totalswap,swapinuse,cpupct)=get_memstats()
        print totalmem,meminuse,totalswap,swapinuse
        lines = [
            'system.%s.pct_mem_used %s %d' % (node, (meminuse/totalmem)*100, timestamp),
            'system.%s.pct_swap_used %s %d' % (node,(swapinuse/totalswap)*100, timestamp),
            'system.%s.pct_cpu_used %s %d' % (node,cpupct, timestamp),
        ]
        message = '\n'.join(lines) + '\n'
        send_msg(message)
      except Exception as e:
        print str(e)
      time.sleep(DELAY)

 

 

 

Leaving ones Comfort Zone (wip).

Written by  on January 9, 2013

The other day another admin was asked to help put together the system uptime reports based on the work done by a long item gone admin who left the organization with a CA-spectrum monitoring legacy. Every now and then uptime reports were created for upper management which consisted of graphs produced from excel spreadsheets which were populated based on spreadsheets produced by the CA suit. The entire process was extremely manual and the problems they were having were with the tools used by any Unix admin to parse data. The questions were about how do I get this text file on to adn off of a non essential unix system so they could run the documented steps to get data from one excel spreadsheet to another. As a long time admin I didnt see much issue with the process since the reports were infrequent and usually “I need this now”. But it seems silly to leave a step by step set of instructions on how to cut, grep, and sed a pile of text to get the requested data for a windows admin. My first thought was just to repeat the process in python but then you have to expect python and its supporting tools to be installed on windows.

So. I started looking at the tools which were already valiable to Windows users and a process that would produce the same

I was surprised to find a fairly rich set in a short time. Now to finish the translation. Then to get the same reports out of the nagios.

Paramiko: python driven ssh.

Written by  on January 4, 2013

As the only unix person available over the holidays the task I was given was to take a list of over a hundred servers and a login (belonging to our manager, a windows sysadmin) that was supposed to have ssh and sudo access and learn as much as I could before the other admin got back from Christmas vacation. Since I have been working primarily in python I started looking at ssh and python and found paramiko which is an ssh implementation built in python. http://www.lag.net/paramiko/docs/paramiko-module.html I was able to use this to determine which servers actually were accessible and which were virtual. Once the other admin was available to fix the accounts that didn’t have access I set about getting a login and password on them. Apparently in the past this took months.

"""
quick script to add my account to the 100+ servers using
my managers login credentials
"""

import paramiko
import getpass
import socket

uname='myboss'
pwd=getpass.getpass()
hosts=["host1",...,"host130"]
for host in hosts:
    try:
        ssh = paramiko.SSHClient()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        ssh.connect(host,username=uname,password=pwd,timeout=3.0)
        stdin,stdout,stderr=ssh.exec_command('/usr/local/bin/sudo /usr/sbin/useradd -u 1119 -g 14 -G"ssh-allow" -c"Donald Davis" -m -d /export/home/ddavis -s /bin/bash ddavis')
        stdin.write(pwd+'\n')
        stdin.flush()
        outlines=stdout.readlines()
        if len(outlines):
            lastout=outlines[0].rstrip('\r\n')
            lastuser=lastout.split(' ',1)[0]
        stdin,stdout,stderr=ssh.exec_command("/usr/bin/grep ddavis /etc/passwd")
        outlines=stdout.readlines()
        if len(outlines):
           print "USER_ADDED::"+host+","+outlines[0].rstrip('\r\n')
        else:
            outlines=stderr.readlines()
            for line in outlines:
                print " ERROR::"+host+","+line.rstrip('\r\n')

        stdin,stdout,stderr=ssh.exec_command("/usr/bin/grep ddavis /etc/group")
        if len(outlines):
           print "GRP_CHANGE::"+host+","+outlines[0].rstrip('\r\n')
        else:
            outlines=stderr.readlines()
            for line in outlines:
                print " ERROR::"+host+","+line.rstrip('\r\n')

    except paramiko.AuthenticationException:
        print " AUTHFAIL::"+host+","+uname
    except Exception as e:
        if "Name or service not known" in str(e):
            print " AWOL::"+host
        elif "timed out" in str(e):
            print " TIMEDOUT::"+host
        else:
            print " UNKNOWN::"+host+",", str(e)

Getting the password set was a little more tricky since passwd actually requires a tty/shell context.

import paramiko
import getpass
import socket
import time

authissues=[]
unknown=[]
awol=[]

uname='myboss'
pwd=getpass.getpass()

hosts=[...listofhosts...]

mypwd=getpass.getpass(0)
for host in hosts:
    try:
        ssh = paramiko.SSHClient()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        ssh.connect(host,username=uname,password=pwd,timeout=1.5)
        chan=ssh.invoke_shell()
        chan.set_combine_stderr(True)
        time.sleep(.6)
        print chan.recv(9999)

        chan.sendall("/usr/local/bin/sudo /usr/bin/passwd ddavis\n")
        time.sleep(.3)
        buff=''
        while 'assword' not in buff:
            resp = chan.recv(9999)
            buff += resp
            print(resp)
            if 'unknown' in buff:
                raise #hell
            if 'denied' in buff:
                raise #hell
            if 'o such file' in buff:
                raise #hell
            if 'not in the sudoers' in buff:
                raise #hell
        chan.send(pwd+'\n')
        time.sleep(.2)

        buff=''
        while 'assword' not in buff:
            resp = chan.recv(9999)
            buff += resp
            print(resp)
            if 'unknown' in buff:
                raise #hell
            if 'denied' in buff:
                raise #hell
            if 'o such file' in buff:
                raise #hell
            if 'not in the sudoers' in buff:
                raise #hell

        chan.send(mypwd+'\n')
        time.sleep(.2)

        buff=''
        while 'assword' not in buff:
            resp = chan.recv(9999)
            buff += resp
            print(resp)
        chan.send(mypwd+'\n')
        time.sleep(.2)

        print chan.recv(2000)
        chan.close()
    except paramiko.AuthenticationException:
        print " AUTHFAIL::"+host+","+uname
        authissues.append(host)
    except Exception as e:
        if "Name or service not known" in str(e):
            print " AWOL::"+host
            awol.append(host)
        elif "timed out" in str(e):
            print " TIMEDOUT::"+host
            awol.append(host)

        else:
            print "UNKNOWN::"+host+",", str(e)
            unknown.append(host)

print "AUTHFAIL", authissues
print "AWOL", awol
print "UNKNOWN",unknown

Finally I can then get rid of the need for passwords altogether with this snippet stolen from the internets.

#!/usr/bin/python
import os
import getpass

authissues=[]
unknown=[]
awol=[]
hosts=[...list of hostnamed ]
import paramiko

def deploy_key(key, server, username, password):
    client = paramiko.SSHClient()
    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    client.connect(server, username=username, password=password)
    client.exec_command('mkdir -p ~/.ssh/')
    client.exec_command('echo "%s" > ~/.ssh/authorized_keys' % key)
    client.exec_command('chmod 644 ~/.ssh/authorized_keys')
    client.exec_command('chmod 700 ~/.ssh/')

key = open(os.path.expanduser('~/.ssh/id_rsa.pub')).read()
username = os.getlogin()
password = getpass.getpass()

for host in hosts:
    try:
        deploy_key(key, host, username, password)
    except paramiko.AuthenticationException:
        print " AUTHFAIL::"+host
        authissues.append(host)
    except Exception as e:
        if "Name or service not known" in str(e):
            print " AWOL::"+host
            awol.append(host)
        elif "timed out" in str(e):
            print " TIMEDOUT::"+host
            awol.append(host)

        else:
            print "UNKNOWN::"+host+",", str(e)
            unknown.append(host)

print "AUTHFAIL", authissues
print "AWOL", awol
print "UNKNOWN",unknown

.

D2G Cricket Settings on CyanogenMod 7.1.0

Written by  on October 31, 2011

I recently installed the CyanogenMod ROM on my Droid 2 Global.  Up to now, I’ve had to take the phone to the Cricket guy to change the settings so that the 3G data connection would work.  Perhaps I could have done it myself then, but I didn’t take the time to figure it out.  Anyway, I can now do it without that occult, crufty, Windows-centric method.  I learned the basics from here:

http://www.droidforums.net/forum/droid-general-discussions/86800-custom-roms-cricket-network.html#post886957

which is a post-flash version of this:

http://www.howardforums.com/showthread.php/1643218-Motorola-Droid-Full-Solution-Updated

I modified the process a bit, though.  So, here’s what I did:

  1. Get http://www.whiterabbit.org/android/u2nl.zip
  2. Push u2nl.zip onto the sdcard either with ADB or USB mount.
  3. Shell into the D2G with ADB or use the Term.app and mount the /system “rw”:
    mount -o rw,remount -t ext3 /dev/block/mmcblk1p21 /system
  4. Unzip the contents of u2nl.zip onto the sdcard:
    cd /sdcard; unzip u2nl.zip
  5. Copy the u2nl executable to /system/bin:
    cp /sdcard/u2nl/u2nl /system/bin/
    chmod 755 /system/bin/u2nl
  6. Create a startup script to execute the commands in autostart.sh:
    cp /sdcard/u2nl/autostart/motodroid/cricket/autostart.sh /etc/init.d/99cricket
    chmod 750 /etc/init.d/99cricket
    chgrp shell /etc/init.d/99cricket
  7. Delete the exit 0 on the last line of the 99cricket script.
  8. Exit the Term.app or the ADB shell.
  9. Menu->Settings->Wireless & Networks->Mobile Networks->Access Point Names
  10. Add the following APN:
    • Name – Cricket
    • APN – internet
    • Proxy – wap.mycricket.com
    • Port – 8080
    • Username – yourphonenumber@mycricket.com
    • Password – cricket
    • Server – wap.mycricket.com
    • MMSC – http://mms.mycricket.com/servlets/mms
    • MMS Proxy – wap.mycricket.com
    • MMS Port – 8080
    • MCC – 310
    • MNC – 004
    • Authentication Type – PAP or CHAP
    • APN type – <Not set>

    And don’t forget to hit Menu->Save.

  11. Reboot

And just for reference, the commands in the autostart.sh/99cricket file are:


#!/system/bin/sh

## set up the iptables for cricket

export PATH="$PATH:/system/bin"
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -F
iptables -t nat -F
iptables -X
iptables -t nat -A OUTPUT -o ppp0 -p 6 ! -d 10.132.25.254 -j REDIRECT --to-port 1025
u2nl 10.132.25.254 8080 127.0.0.1 1025 >/dev/null 2>&1 &
sh -c "sleep 5;kill `ps|grep nk.bla.android.autostart|grep -v grep|awk '{print $2}'`" &

Thanks to http://www.whiterabbit.org/android/.

Axiom of Regularity and AFA theorem decidability

Written by  on April 5, 2011

Below are some thoughts I had in relation to an e-mail discussion where the following question arose: What is a representation? The arching context of this discussion was a seminar exploring the psychological theory that feelings are a form of self-perception. I.e. if self-perception theory is true, this would explain the abundance of clinical evidence showing that feelings follow behaviors rather than preceding them.

In any case, the group has a large contingent of technical types and the discussion of representation eventually raised the question of the difference between processes and things, verbs and nouns, [en|de]coders and an encoding, or code and data. The underlying itch seems to be the concept of self-modifying processes or, in computer science terms, code as data and data as code. It seemed to me the holistic thinkers see an aspect to living things, organisms, that is not present in machines. (This inarticulate distinction isn’t new, of course. Great minds have been trying to tease them apart for millenia. And my puny mind won’t add anything to what they’ve already done. But this blog has gotten a bit stale; so I must post something.)

The group seemed to have stalled at the ontological status of a representation. Such a thing seems to have an autonomy all on its own. For example, if you scratch an image of your grandmother into a piece of granite, then humanity goes extinct, do those scratches on the granite still represent your grandmother? Or are they then just meaningless scratches on a rock? To what extent is the decoding process required for an encoded artifact to still be meaningful?

Anyway, this obviously raises all sorts of questions for anyone who likes to think deeply or completely. But the (shallow and incomplete) question it raises to me is fairly common to all the philosophical problems I think about: circular reference. In logic, an impredicative definition is one that defines an object using a quantification over all the other objects like the object being defined. In other words, it’s a self-referencing definition. It’s im-predicative simply because it’s not predicative. In the foundations of math, it boils down to von Neumann’s Axiom of Regularity, which states that sets cannot have elements that, when cracked open, contain elements present anywhere in the cumulative hierarchy of sets above them. In other words, the set hierarchy cannot contain loops.

It seems to me that an organism, and in particular those of us who are capable of using symbols to communicate with one another, is impredicatively defined in terms of its environment. The symbols used are in constant redefinition as the organism and its environment evolve. Hence, any representation, e.g. of the environment by the organism or of the organism by the environment, will be self-referential and inseparable from its context.

So, this leads me to ask why we ascribe ontological status to representations? Why do we think the image of a grandmother scratched into a rock will persist as a representation? I think the answer lies in how we count, which is (probably) based on the fact that we have fingers, digits. We count ordinally: 1, 2, 3 …. And because we do so, our intuitive concept of numbers and sets of things is constructive. Abstractly, math is about grammar, sentences, consistency, and completeness. But concretely, math is about how we relate large bunches of things to small bunches of things, 100 cows to 10 bails of hay, 10 fingers to a billion stars, etc.

When considering these relationships, abstract or concrete, various problems arise like how to represent nothing, the result of taking away 6 cows from a total of 6 cows to give you no cows. How does one represent no cows? Infinity and density are other such problems. What lies after the largest number to which you can count? What lies between two very close numbers? Etc.

In the course of handling some of these issues, von Neumann formulated a constraint (regularity) to avoid unconstructable, cyclic set hierarchies. Later, Turing came up with the “halting problem”: given a step-by-step procedure, decide whether the procedure halts or continues forever. These sorts of problem obviously depend critically on construction. Whether you can tell if a procedure will halt depends on whether and how it can be constructed.

It is this sense of constructability and automatic deduction (no consciousness required), that I think leads to ascribing more ontological status to representations/encodings than is objectively warranted. Another interesting twist arises because so many people use the computer as a metaphor for organisms, the hardware is likened to the body and the software is likened to the mind. But again, this metaphor raises the hackles of those who think organisms are categorically different from machines.

We can seamlessly use the term “representation” to mean a thought in the mind of an organism and an encoding in the software of a machine. But the character of the term changes between the two usages. Saying “the computer uses a representation of your grandmother” is very different to an ordinary layperson than saying “you have a representation of your grandmother in your mind”.

Perhaps this goes back to the axiom of regularity in that the most common modern computer architecture is the von Neumann architecture, where the CPU (process) is categorically distinct from the memory (objects) upon which it operates. These modern computers function by counting, constructing one state from previous states. That separation between the CPU and the memory disallows cyclic sets. At any given time, the objects in the memory can be enumerated, which would not be true without the axiom of regularity. No register can be a member of itself.

Of course, we can simulate circularity by first enumerating some of the set, then moving the CPU pointer back to a register it already finished reading and letting it read again from that same register. But this is not a circular or impredicative definition because at any particular time, the memory is completely, acyclically, enumerable. I take this to imply that a von Neumann architecture is incapable of realizing a non-well-founded set.

Of course, I’m not really a mathematician and I’m certainly not a meta-mathematician. So, my intuition and/or reasoning could be completely off. One discussant pointed out that an alternative set of axioms, called AFA (Anti-Foundation Axiom, which refers to the ZFC without the axiom of regularity plus the anti-foundation axiom) is just as consistent as ZFC (Zermelo-Fraenkel plus the Axiom of Choice). And others[1] point out that AFA and ZFC are shown to be “mutually interpretable”. So, perhaps this means that a sentence in AFA, involving a cyclic set (not possible in ZFC), can be “constructed” by first transforming the sentence into some equivalent in ZFC, constructing that in the normal way, then transforming it back. This would mean that a von Neumann architecture machine, although based in the intuitive counting-based construction with which we’re familiar, would still be able to compute AFA sentences … kinda like transforming a problem from the time domain into the frequency domain, solving it there, and transforming it back. I don’t know. But it sure seems suspicious to me.

In any case, perhaps my suspicion is evidence that I’ve bought into the false dichotomy von Neumann so insidiously implanted in our modern, digitally myopic minds? Perhaps representations (or symbols, in general) really do become entirely meaningless when violently torn from their context?

[1] http://research.microsoft.com/pubs/70350/tr-2006-138.pdf

Plain text message composition citation color in Thunderbird

Written by  on February 24, 2011

Because it took me awhile to find this, I want to document it here. You can change much of the preferences in Thunderbird through the GUI. And you can change them directly with the Edit -> Preferences -> Advanced -> General -> Config Editor. But there doesn’t seem to be a way to set the color of the cited text in a plain text reply. In order to change that, you have to edit/create this file:

/home/[username]/.mozilla-thunderbird/[profile].default/chrome/userContent.css

In that file, place the following CSS clause:

span[_moz_quote=true] {
color: pink ! important;
}

But shut Thunderbird down before you edit that file. Thanks to this forum thread for the answer.

Ubuntu Maverick (10.10) EC2 images

Written by  on January 20, 2011

There were two obscure tricks to getting new EC2 Ubuntu Maverick (10.10) images for running our In Silico Liver Simulation:

  1. change /boot/grub/menu.lst root=LABEL=uec-rootfs to root=/dev/sda1, and
  2. copying some kernel modules so that nfs-kernel-server will work. link

Expanding an VirtualBox XP system disk

Written by  on September 1, 2010

After spending an extraordinary amount of time trying to figure out why I couldn’t get NTFSResize to work, I finally found Seppe vanden Broucke’s log entry. For posterity, I’ll record what I did, here, following partly the instructions in the original post and one commenter’s modification:

  1. Get the SysRescCD ISO image. (Any live linux CD should work, including Knoppix.)
  2. Create a new (larger) VirtualBox disk image (*.vdi) from within VirtualBox.
  3. In the VirtualBox Details tab for your Windows instance, set your original VDI as the IDE Primary Master.
  4. Set the new (empty) VDI as the IDE Primary Slave.
  5. And set the linux ISO as the CD/DVD IDE Secondary Master.
  6. Ensure that the instance is set to boot from the CD/DVD first in System->Boot Order.
  7. Boot the instance.
  8. At the command line, run
    fdisk -l /dev/sda
    You should see a partition there of type 7, NTFS.
  9. Then run
    fdisk -l /dev/sdb
    You should see an error message that says there is no partition on that device.
  10. Then run the command:
    dd if=/dev/sda of=/dev/sdb
    This will take a long time. It means copy from the input file /dev/sda to the output file /dev/sdb.
  11. Shutdown the system with:
    shutdown -h now
  12. Change the Storage->IDE Secondary Master (CD/DVD) from the ISO image to your regular optical drive, or remove it completely so that it will boot into the original (small) Windows image.
  13. In Windows, select Start->Run… and type:
    diskpart.exe
  14. Type:
    select disk 1
    to select the (large) disk.
  15. Type:
    list disk
    to see that you’ve selected the right one with the right size.
  16. Type:
    select partition 1
    to select the 1st partition on that disk.
  17. Type:
    list partition
    to see that you’ve selected the right partition.
  18. Type:
    extend
    to extend the volume all the way to the end of the device.
  19. Turn off the virtual instance and select the new (larger) VDI as the Storage->IDE Primary Master.
  20. Boot the instance again and you should have a larger system (C:) disk.