http://mpastell.com/ mpastell.com - Articles (Atom Feed) 2013-05-01T21:00:00Z Matti Pastell http://mpastell.com tag:mpastell.com,2013-05-02:/2013/05/02/matplotlib_colormaps/ Matplotlib colormaps with a surface plot 2013-05-01T21:00:00Z 2013-05-01T21:00:00Z <p> Today I was working on surface plots and wanted to know what different colormaps in <a href="http://matplotlib.org/">Matplotlib</a> look like. Luckily it was easy to find out and I decided to post results here in case someone else finds it useful. </p> <p> Below you’ll find a surface plot of a multivariate normal distribution plotted with every (n=140) colormap in Matplotlib 1.2.0. The name of the colormap is the title of each plot. This page was created using <a href="http://mpastell.com/pweave">Pweave</a>. </p> <div class="highlight"> <pre><span class="kn">from</span> <span class="nn">pylab</span> <span class="kn">import</span> <span class="o">*</span> <span class="kn">from</span> <span class="nn">mpl_toolkits.mplot3d</span> <span class="kn">import</span> <span class="n">Axes3D</span> <span class="n">x</span> <span class="o">=</span> <span class="n">linspace</span><span class="p">(</span><span class="o">-</span><span class="mi">5</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">200</span><span class="p">)</span> <span class="n">y</span> <span class="o">=</span> <span class="n">x</span> <span class="n">X</span><span class="p">,</span><span class="n">Y</span> <span class="o">=</span> <span class="n">meshgrid</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span> <span class="n">Z</span> <span class="o">=</span> <span class="n">bivariate_normal</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Y</span><span class="p">)</span> <span class="k">for</span> <span class="n">cmap</span> <span class="ow">in</span> <span class="n">colormaps</span><span class="p">():</span> <span class="n">fig</span> <span class="o">=</span> <span class="n">figure</span><span class="p">()</span> <span class="n">ax</span> <span class="o">=</span> <span class="n">fig</span><span class="o">.</span><span class="n">add_subplot</span><span class="p">(</span><span class="mi">111</span><span class="p">,</span> <span class="n">projection</span><span class="o">=</span><span class="s">&#39;3d&#39;</span><span class="p">)</span> <span class="n">ax</span><span class="o">.</span><span class="n">plot_surface</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Y</span><span class="p">,</span> <span class="n">Z</span><span class="p">,</span> <span class="n">cmap</span><span class="o">=</span> <span class="n">cmap</span><span class="p">)</span> <span class="n">title</span><span class="p">(</span><span class="n">cmap</span><span class="p">)</span> </pre> </div> <p><img src="figures/bi_normal_figure1_1.jpg" width="460"/> <img src="figures/bi_normal_figure1_2.jpg" width="460"/> <img src="figures/bi_normal_figure1_3.jpg" width="460"/> <img src="figures/bi_normal_figure1_4.jpg" width="460"/> <img src="figures/bi_normal_figure1_5.jpg" width="460"/> <img src="figures/bi_normal_figure1_6.jpg" width="460"/> <img src="figures/bi_normal_figure1_7.jpg" width="460"/> <img src="figures/bi_normal_figure1_8.jpg" width="460"/> <img src="figures/bi_normal_figure1_9.jpg" width="460"/> <img src="figures/bi_normal_figure1_10.jpg" width="460"/> <img src="figures/bi_normal_figure1_11.jpg" width="460"/> <img src="figures/bi_normal_figure1_12.jpg" width="460"/> <img src="figures/bi_normal_figure1_13.jpg" width="460"/> <img src="figures/bi_normal_figure1_14.jpg" width="460"/> <img src="figures/bi_normal_figure1_15.jpg" width="460"/> <img src="figures/bi_normal_figure1_16.jpg" width="460"/> <img src="figures/bi_normal_figure1_17.jpg" width="460"/> <img src="figures/bi_normal_figure1_18.jpg" width="460"/> <img src="figures/bi_normal_figure1_19.jpg" width="460"/> <img src="figures/bi_normal_figure1_20.jpg" width="460"/> <img src="figures/bi_normal_figure1_21.jpg" width="460"/> <img src="figures/bi_normal_figure1_22.jpg" width="460"/> <img src="figures/bi_normal_figure1_23.jpg" width="460"/> <img src="figures/bi_normal_figure1_24.jpg" width="460"/> <img src="figures/bi_normal_figure1_25.jpg" width="460"/> <img src="figures/bi_normal_figure1_26.jpg" width="460"/> <img src="figures/bi_normal_figure1_27.jpg" width="460"/> <img src="figures/bi_normal_figure1_28.jpg" width="460"/> <img src="figures/bi_normal_figure1_29.jpg" width="460"/> <img src="figures/bi_normal_figure1_30.jpg" width="460"/> <img src="figures/bi_normal_figure1_31.jpg" width="460"/> <img src="figures/bi_normal_figure1_32.jpg" width="460"/> <img src="figures/bi_normal_figure1_33.jpg" width="460"/> <img src="figures/bi_normal_figure1_34.jpg" width="460"/> <img src="figures/bi_normal_figure1_35.jpg" width="460"/> <img src="figures/bi_normal_figure1_36.jpg" width="460"/> <img src="figures/bi_normal_figure1_37.jpg" width="460"/> <img src="figures/bi_normal_figure1_38.jpg" width="460"/> <img src="figures/bi_normal_figure1_39.jpg" width="460"/> <img src="figures/bi_normal_figure1_40.jpg" width="460"/> <img src="figures/bi_normal_figure1_41.jpg" width="460"/> <img src="figures/bi_normal_figure1_42.jpg" width="460"/> <img src="figures/bi_normal_figure1_43.jpg" width="460"/> <img src="figures/bi_normal_figure1_44.jpg" width="460"/> <img src="figures/bi_normal_figure1_45.jpg" width="460"/> <img src="figures/bi_normal_figure1_46.jpg" width="460"/> <img src="figures/bi_normal_figure1_47.jpg" width="460"/> <img src="figures/bi_normal_figure1_48.jpg" width="460"/> <img src="figures/bi_normal_figure1_49.jpg" width="460"/> <img src="figures/bi_normal_figure1_50.jpg" width="460"/> <img src="figures/bi_normal_figure1_51.jpg" width="460"/> <img src="figures/bi_normal_figure1_52.jpg" width="460"/> <img src="figures/bi_normal_figure1_53.jpg" width="460"/> <img src="figures/bi_normal_figure1_54.jpg" width="460"/> <img src="figures/bi_normal_figure1_55.jpg" width="460"/> <img src="figures/bi_normal_figure1_56.jpg" width="460"/> <img src="figures/bi_normal_figure1_57.jpg" width="460"/> <img src="figures/bi_normal_figure1_58.jpg" width="460"/> <img src="figures/bi_normal_figure1_59.jpg" width="460"/> <img src="figures/bi_normal_figure1_60.jpg" width="460"/> <img src="figures/bi_normal_figure1_61.jpg" width="460"/> <img src="figures/bi_normal_figure1_62.jpg" width="460"/> <img src="figures/bi_normal_figure1_63.jpg" width="460"/> <img src="figures/bi_normal_figure1_64.jpg" width="460"/> <img src="figures/bi_normal_figure1_65.jpg" width="460"/> <img src="figures/bi_normal_figure1_66.jpg" width="460"/> <img src="figures/bi_normal_figure1_67.jpg" width="460"/> <img src="figures/bi_normal_figure1_68.jpg" width="460"/> <img src="figures/bi_normal_figure1_69.jpg" width="460"/> <img src="figures/bi_normal_figure1_70.jpg" width="460"/> <img src="figures/bi_normal_figure1_71.jpg" width="460"/> <img src="figures/bi_normal_figure1_72.jpg" width="460"/> <img src="figures/bi_normal_figure1_73.jpg" width="460"/> <img src="figures/bi_normal_figure1_74.jpg" width="460"/> <img src="figures/bi_normal_figure1_75.jpg" width="460"/> <img src="figures/bi_normal_figure1_76.jpg" width="460"/> <img src="figures/bi_normal_figure1_77.jpg" width="460"/> <img src="figures/bi_normal_figure1_78.jpg" width="460"/> <img src="figures/bi_normal_figure1_79.jpg" width="460"/> <img src="figures/bi_normal_figure1_80.jpg" width="460"/> <img src="figures/bi_normal_figure1_81.jpg" width="460"/> <img src="figures/bi_normal_figure1_82.jpg" width="460"/> <img src="figures/bi_normal_figure1_83.jpg" width="460"/> <img src="figures/bi_normal_figure1_84.jpg" width="460"/> <img src="figures/bi_normal_figure1_85.jpg" width="460"/> <img src="figures/bi_normal_figure1_86.jpg" width="460"/> <img src="figures/bi_normal_figure1_87.jpg" width="460"/> <img src="figures/bi_normal_figure1_88.jpg" width="460"/> <img src="figures/bi_normal_figure1_89.jpg" width="460"/> <img src="figures/bi_normal_figure1_90.jpg" width="460"/> <img src="figures/bi_normal_figure1_91.jpg" width="460"/> <img src="figures/bi_normal_figure1_92.jpg" width="460"/> <img src="figures/bi_normal_figure1_93.jpg" width="460"/> <img src="figures/bi_normal_figure1_94.jpg" width="460"/> <img src="figures/bi_normal_figure1_95.jpg" width="460"/> <img src="figures/bi_normal_figure1_96.jpg" width="460"/> <img src="figures/bi_normal_figure1_97.jpg" width="460"/> <img src="figures/bi_normal_figure1_98.jpg" width="460"/> <img src="figures/bi_normal_figure1_99.jpg" width="460"/> <img src="figures/bi_normal_figure1_100.jpg" width="460"/> <img src="figures/bi_normal_figure1_101.jpg" width="460"/> <img src="figures/bi_normal_figure1_102.jpg" width="460"/> <img src="figures/bi_normal_figure1_103.jpg" width="460"/> <img src="figures/bi_normal_figure1_104.jpg" width="460"/> <img src="figures/bi_normal_figure1_105.jpg" width="460"/> <img src="figures/bi_normal_figure1_106.jpg" width="460"/> <img src="figures/bi_normal_figure1_107.jpg" width="460"/> <img src="figures/bi_normal_figure1_108.jpg" width="460"/> <img src="figures/bi_normal_figure1_109.jpg" width="460"/> <img src="figures/bi_normal_figure1_110.jpg" width="460"/> <img src="figures/bi_normal_figure1_111.jpg" width="460"/> <img src="figures/bi_normal_figure1_112.jpg" width="460"/> <img src="figures/bi_normal_figure1_113.jpg" width="460"/> <img src="figures/bi_normal_figure1_114.jpg" width="460"/> <img src="figures/bi_normal_figure1_115.jpg" width="460"/> <img src="figures/bi_normal_figure1_116.jpg" width="460"/> <img src="figures/bi_normal_figure1_117.jpg" width="460"/> <img src="figures/bi_normal_figure1_118.jpg" width="460"/> <img src="figures/bi_normal_figure1_119.jpg" width="460"/> <img src="figures/bi_normal_figure1_120.jpg" width="460"/> <img src="figures/bi_normal_figure1_121.jpg" width="460"/> <img src="figures/bi_normal_figure1_122.jpg" width="460"/> <img src="figures/bi_normal_figure1_123.jpg" width="460"/> <img src="figures/bi_normal_figure1_124.jpg" width="460"/> <img src="figures/bi_normal_figure1_125.jpg" width="460"/> <img src="figures/bi_normal_figure1_126.jpg" width="460"/> <img src="figures/bi_normal_figure1_127.jpg" width="460"/> <img src="figures/bi_normal_figure1_128.jpg" width="460"/> <img src="figures/bi_normal_figure1_129.jpg" width="460"/> <img src="figures/bi_normal_figure1_130.jpg" width="460"/> <img src="figures/bi_normal_figure1_131.jpg" width="460"/> <img src="figures/bi_normal_figure1_132.jpg" width="460"/> <img src="figures/bi_normal_figure1_133.jpg" width="460"/> <img src="figures/bi_normal_figure1_134.jpg" width="460"/> <img src="figures/bi_normal_figure1_135.jpg" width="460"/> <img src="figures/bi_normal_figure1_136.jpg" width="460"/> <img src="figures/bi_normal_figure1_137.jpg" width="460"/> <img src="figures/bi_normal_figure1_138.jpg" width="460"/> <img src="figures/bi_normal_figure1_139.jpg" width="460"/> <img src="figures/bi_normal_figure1_140.jpg" width="460"/></p> tag:mpastell.com,2013-04-19:/2013/04/19/python_regression/ Linear Regression Models with Python 2013-04-18T21:00:00Z 2013-04-18T21:00:00Z <p>I have been looking into using Python for basic statistical analyses lately and I decided to write a short example about fitting linear regression models using <a href="http://statsmodels.sourceforge.net/devel/generated/statsmodels.stats.outliers_influence.OLSInfluence.html">statsmodels</a>-library.</p> <h1> Requirements </h1> <p> This example uses <a href="http://statsmodels.sourceforge.net/devel/">statsmodels</a> version 0.5 from github and we’ll use the new formula API which makes fitting the models very familiar for R users. You’ll also need <a href="http://www.numpy.org/">Numpy</a>, <a href="http://pandas.pydata.org/">Pandas</a> and <a href="http://matplotlib.org/">matplolib</a>. </p> <p> <p>The analysis has been published using <a href="http://mpastell.com/pweave/">Pweave development version</a>. See my <a href="/2013/04/19/publish_python_scripts/">other post</a>.</p> <p> Import libraries </p> <div class="highlight"> <pre><span class="kn">import</span> <span class="nn">pandas</span> <span class="kn">as</span> <span class="nn">pd</span> <span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span> <span class="kn">import</span> <span class="nn">statsmodels.formula.api</span> <span class="kn">as</span> <span class="nn">sm</span> <span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="kn">as</span> <span class="nn">plt</span> </pre> </div> <p> We’ll use <a href="http://stat.ethz.ch/R-manual/R-patched/library/MASS/html/whiteside.html">whiteside</a> dataset from R package MASS. You can read the description of the dataset from the link, but in short it contains: </p> <blockquote> <p> <em>The weekly gas consumption and average external temperature at a house in south-east England for two heating seasons, one of 26 weeks before, and one of 30 weeks after cavity-wall insulation was installed.</em> </p> </blockquote> <p> Read the data from <a href="https://github.com/cpcloud/pydatasets">pydatasets repo</a> using Pandas: </p> <div class="highlight"> <pre><span class="n">url</span> <span class="o">=</span> <span class="s">&#39;https://raw.github.com/cpcloud/pydatasets/master/datasets/MASS/whiteside.csv&#39;</span> <span class="n">whiteside</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_csv</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> </pre> </div> <h1> Fitting the model </h1> <p> Let’s see what the relationship between the gas consumption is before the insulation. See <a href="http://statsmodels.sourceforge.net/devel/example_formulas.html">statsmodels documentation</a> for more information about the syntax. </p> <div class="highlight"> <pre><span class="n">model</span> <span class="o">=</span> <span class="n">sm</span><span class="o">.</span><span class="n">ols</span><span class="p">(</span><span class="n">formula</span><span class="o">=</span><span class="s">&#39;Gas ~ Temp&#39;</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">whiteside</span><span class="p">,</span> <span class="n">subset</span> <span class="o">=</span> <span class="n">whiteside</span><span class="p">[</span><span class="s">&#39;Insul&#39;</span><span class="p">]</span><span class="o">==</span><span class="s">&quot;Before&quot;</span><span class="p">)</span> <span class="n">fitted</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">()</span> <span class="k">print</span> <span class="n">fitted</span><span class="o">.</span><span class="n">summary</span><span class="p">()</span> </pre> </div> <div class="highlight"> <pre> <span class="n">OLS</span> <span class="n">Regression</span> <span class="n">Results</span> <span class="o">==============================================================================</span> <span class="n">Dep</span><span class="o">.</span> <span class="n">Variable</span><span class="p">:</span> <span class="n">Gas</span> <span class="n">R</span><span class="o">-</span><span class="n">squared</span><span class="p">:</span> <span class="mf">0.944</span> <span class="n">Model</span><span class="p">:</span> <span class="n">OLS</span> <span class="n">Adj</span><span class="o">.</span> <span class="n">R</span><span class="o">-</span><span class="n">squared</span><span class="p">:</span> <span class="mf">0.941</span> <span class="n">Method</span><span class="p">:</span> <span class="n">Least</span> <span class="n">Squares</span> <span class="n">F</span><span class="o">-</span><span class="n">statistic</span><span class="p">:</span> <span class="mf">403.1</span> <span class="n">Date</span><span class="p">:</span> <span class="n">Fri</span><span class="p">,</span> <span class="mi">19</span> <span class="n">Apr</span> <span class="mi">2013</span> <span class="n">Prob</span> <span class="p">(</span><span class="n">F</span><span class="o">-</span><span class="n">statistic</span><span class="p">):</span> <span class="mf">1.64e-16</span> <span class="n">Time</span><span class="p">:</span> <span class="mi">16</span><span class="p">:</span><span class="mo">07</span><span class="p">:</span><span class="mi">56</span> <span class="n">Log</span><span class="o">-</span><span class="n">Likelihood</span><span class="p">:</span> <span class="o">-</span><span class="mf">2.8783</span> <span class="n">No</span><span class="o">.</span> <span class="n">Observations</span><span class="p">:</span> <span class="mi">26</span> <span class="n">AIC</span><span class="p">:</span> <span class="mf">9.757</span> <span class="n">Df</span> <span class="n">Residuals</span><span class="p">:</span> <span class="mi">24</span> <span class="n">BIC</span><span class="p">:</span> <span class="mf">12.27</span> <span class="n">Df</span> <span class="n">Model</span><span class="p">:</span> <span class="mi">1</span> <span class="o">==============================================================================</span> <span class="n">coef</span> <span class="n">std</span> <span class="n">err</span> <span class="n">t</span> <span class="n">P</span><span class="o">&gt;|</span><span class="n">t</span><span class="o">|</span> <span class="p">[</span><span class="mf">95.0</span><span class="o">%</span> <span class="n">Conf</span><span class="o">.</span> <span class="n">Int</span><span class="o">.</span><span class="p">]</span> <span class="o">------------------------------------------------------------------------------</span> <span class="n">Intercept</span> <span class="mf">6.8538</span> <span class="mf">0.118</span> <span class="mf">57.876</span> <span class="mf">0.000</span> <span class="mf">6.609</span> <span class="mf">7.098</span> <span class="n">Temp</span> <span class="o">-</span><span class="mf">0.3932</span> <span class="mf">0.020</span> <span class="o">-</span><span class="mf">20.078</span> <span class="mf">0.000</span> <span class="o">-</span><span class="mf">0.434</span> <span class="o">-</span><span class="mf">0.353</span> <span class="o">==============================================================================</span> <span class="n">Omnibus</span><span class="p">:</span> <span class="mf">0.296</span> <span class="n">Durbin</span><span class="o">-</span><span class="n">Watson</span><span class="p">:</span> <span class="mf">2.420</span> <span class="n">Prob</span><span class="p">(</span><span class="n">Omnibus</span><span class="p">):</span> <span class="mf">0.862</span> <span class="n">Jarque</span><span class="o">-</span><span class="n">Bera</span> <span class="p">(</span><span class="n">JB</span><span class="p">):</span> <span class="mf">0.164</span> <span class="n">Skew</span><span class="p">:</span> <span class="o">-</span><span class="mf">0.177</span> <span class="n">Prob</span><span class="p">(</span><span class="n">JB</span><span class="p">):</span> <span class="mf">0.921</span> <span class="n">Kurtosis</span><span class="p">:</span> <span class="mf">2.839</span> <span class="n">Cond</span><span class="o">.</span> <span class="n">No</span><span class="o">.</span> <span class="mf">13.3</span> <span class="o">==============================================================================</span> </pre> </div> <h1> Plot the data and fit </h1> <div class="highlight"> <pre><span class="n">Before</span> <span class="o">=</span> <span class="n">whiteside</span><span class="p">[</span><span class="n">whiteside</span><span class="p">[</span><span class="s">&quot;Insul&quot;</span><span class="p">]</span> <span class="o">==</span> <span class="s">&quot;Before&quot;</span><span class="p">]</span> <span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">Before</span><span class="p">[</span><span class="s">&quot;Temp&quot;</span><span class="p">],</span> <span class="n">Before</span><span class="p">[</span><span class="s">&quot;Gas&quot;</span><span class="p">],</span> <span class="s">&#39;ro&#39;</span><span class="p">)</span> <span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">Before</span><span class="p">[</span><span class="s">&quot;Temp&quot;</span><span class="p">],</span> <span class="n">fitted</span><span class="o">.</span><span class="n">fittedvalues</span><span class="p">,</span> <span class="s">&#39;b&#39;</span><span class="p">)</span> <span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">([</span><span class="s">&#39;Data&#39;</span><span class="p">,</span> <span class="s">&#39;Fitted model&#39;</span><span class="p">])</span> <span class="n">plt</span><span class="o">.</span><span class="n">ylim</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="n">plt</span><span class="o">.</span><span class="n">xlim</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="mi">12</span><span class="p">)</span> <span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s">&#39;Temperature&#39;</span><span class="p">)</span> <span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s">&#39;Gas&#39;</span><span class="p">)</span> <span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s">&#39;Before Insulation&#39;</span><span class="p">)</span> </pre> </div> <p> <img src="figures/regression_whiteside_figure4_1.png" width="600"/> </p> <h1> Fit diagnostiscs </h1> <p> Statsmodels <a href="http://statsmodels.sourceforge.net/devel/generated/statsmodels.regression.linear_model.OLSResults.html">OLSresults</a> objects contain the usual diagnostic information about the model and you can use the <code>get_influence()</code> method to get more diagnostic information (such as Cook’s distance). </p> <h2> A look at the residuals </h2> <p> Histogram of normalized residuals </p> <div class="highlight"> <pre><span class="n">plt</span><span class="o">.</span><span class="n">hist</span><span class="p">(</span><span class="n">fitted</span><span class="o">.</span><span class="n">norm_resid</span><span class="p">())</span> <span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s">&#39;Count&#39;</span><span class="p">)</span> <span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s">&#39;Normalized residuals&#39;</span><span class="p">)</span> </pre> </div> <p> <img src="figures/regression_whiteside_figure5_1.png" width="600"/> </p> <h2> Cooks distance </h2> <p> <a href="http://statsmodels.sourceforge.net/devel/generated/statsmodels.stats.outliers_influence.OLSInfluence.html">OLSInfluence</a> objects contain more diagnostic information </p> <div class="highlight"> <pre><span class="n">influence</span> <span class="o">=</span> <span class="n">fitted</span><span class="o">.</span><span class="n">get_influence</span><span class="p">()</span> <span class="c">#c is the distance and p is p-value</span> <span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">p</span><span class="p">)</span> <span class="o">=</span> <span class="n">influence</span><span class="o">.</span><span class="n">cooks_distance</span> <span class="n">plt</span><span class="o">.</span><span class="n">stem</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">c</span><span class="p">)),</span> <span class="n">c</span><span class="p">,</span> <span class="n">markerfmt</span><span class="o">=</span><span class="s">&quot;,&quot;</span><span class="p">)</span> </pre> </div> <p> <img src="figures/regression_whiteside_figure6_1.png" width="600"/> </p> <h1> Statsmodels builtin plots </h1> <p> Statsmodels includes a some builtin function for plotting residuals against leverage: </p> <div class="highlight"> <pre><span class="kn">from</span> <span class="nn">statsmodels.graphics.regressionplots</span> <span class="kn">import</span> <span class="o">*</span> <span class="n">plot_leverage_resid2</span><span class="p">(</span><span class="n">fitted</span><span class="p">)</span> <span class="n">influence_plot</span><span class="p">(</span><span class="n">fitted</span><span class="p">)</span> </pre> </div> <p> <img src="figures/regression_whiteside_figure7_1.png" width="600"/> <img src="figures/regression_whiteside_figure7_2.png" width="600"/> </p> tag:mpastell.com,2013-04-19:/2013/04/19/publish_python_scripts/ Publish Python Scripts with Pweave 2013-04-18T21:00:00Z 2013-04-18T21:00:00Z <p>I have been working on <a href="http://mpastell.com/pweave">Pweave</a> lately and one new feature for the next release is a new input format and a HTML template that allows easy publishing of regular python scripts. The feature is similar to MATLAB’s publish and the input format for doc and code chunks has been adopted from <a href="http://yihui.name/knitr/">knitr</a>’s <a href="http://yihui.name/knitr/demo/stitch/">spin</a>. You can use the new format instead of the old noweb format for any document, one advantage is that you can also run the code without Pweave as a normal Python script which makes debugging easier.</p> <h2 id="have-a-look-at-couple-of-examples-about-published-scipts">Have a look at couple of examples about published scipts:</h2> <p><strong>FIR filter design</strong> <a href="http://files.mpastell.com/published/FIR_design.html">html</a>, <a href="http://files.mpastell.com/published/FIR_design.pdf">pdf</a> and <a href="http://files.mpastell.com/published/FIR_design.py">code</a>. Also explains the basics of the format.<br /> <strong>Linear regression</strong> <a href="http://files.mpastell.com/published/regression_whiteside.html">html</a>, <a href="http://files.mpastell.com/published/regression_whiteside.pdf">pdf</a> and <a href="http://files.mpastell.com/published/regression_whiteside.py">code</a>. Linear regression using statsmodels.</p> <h2 id="try-it-out">Try it out</h2> <p>I’d appreciate any feedback before the next version is released, so drop me a mail if you have any thoughts.</p> <p>Publishing scripts is easy: Get the current development version from <a href="https://bitbucket.org/mpastell/pweave">Bitbucket repo</a> and install it. You’ll also need <a href="https://pypi.python.org/pypi/Markdown">python-markdown</a> and for <a href="https://pypi.python.org/pypi/markdown2Mathjax/0.3.8">markdown2Mathjax</a> to use inline math.</p> <p>Publish a script to html from the command line.</p> <div class="highlight"> <pre>pypublish FIR_design.py</pre> </div> <p>And to pdf (requires pandoc and pdflatex):</p> <div class="highlight"> <pre>pypublish -f pdf FIR_design.py</pre> </div> tag:mpastell.com,2010-03-25:/2010/03/25/create-odf-pdf-and-html-report-from-a-single-sweave-document/ Create odf, pdf and html report from a single Sweave document 2010-03-24T22:00:00Z 2010-03-24T22:00:00Z <p>A lot of us know about Sweave and Latex and they work very well in creating elegant dynamic reports from R computation. However, sometimes we would like to also produce a word processing document for a colleague or a html version of the same report. Now there are tools for producing these like <a href="http://cran.r-project.org/web/packages/odfWeave/index.html">odfWeave</a>. But wouldn’t it be nice to get all of the output formats from the source document with same mark up?</p> <p>Today I’m going to write about a good option how to get a <em>nice looking</em> odf, pdf and html report from the same source: ascii package with <a href="http://docutils.sourceforge.net/rst.html">reStructuredText</a> (reST) markup. reStructuredText is a simple plain text mark up that can be converted to several formats using the Python <a href="http://docutils.sourceforge.net/index.html">Docutils</a>. It is simpler to write than Latex and it also the syntax that is used in <a href="http://sphinx.pocoo.org">Sphinx</a> documents. The <a href="http://eusebe.github.com/ascii/">ascii</a> package has a Sweave driver for weaving R code from documents written with reST and some other mark up languages as well. I have created a simple example document <a href="http://files.mpastell.com/ascii/ascii-example.Rnw">ascii-example.Rnw</a>, which demonstrates the capabilities of the combination. The source file is a Sweave document with reST mark up instead of Latex. I had to make small modifications to the in ReST driver in ascii package to get this example to work<a href="#fn1" class="footnoteRef" id="fnref1"><sup>1</sup></a> (download it here: <a href="http://files.mpastell.com/ascii/newRest.R">newRest.R</a>). I then processed the example with the new driver in R:</p> <div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">&gt;<span class="st"> </span><span class="kw">library</span>(ascii) &gt;<span class="st"> </span><span class="kw">source</span>(<span class="st">&#39;newRest.R&#39;</span>) &gt;<span class="st"> </span><span class="kw">ReST</span>(<span class="st">&#39;ascii-example.Rnw&#39;</span>)</code></pre></div> <p>Weaving produced this <a href="http://files.mpastell.com/ascii/ascii-example.rst">ascii-example.rst</a> reST document which I then converted to odf, html, pdf with pdflatex and pdf with <a href="http://code.google.com/p/rst2pdf/">rst2pdf</a> using:</p> <div class="sourceCode"><pre class="sourceCode bash"><code class="sourceCode bash"><span class="kw">rst2odt</span> ascii-example.rst ascii-example.odt <span class="kw">rst2html</span> ascii-example.rst ascii-example.html <span class="kw">rst2latex</span> ascii-example.rst ascii-example.tex <span class="kw">&amp;&amp;</span> <span class="kw">pdflatex</span> ascii-example.tex <span class="kw">rst2pdf</span> ascii-example.rst</code></pre></div> <p><strong>And here are the results:</strong> <a href="http://files.mpastell.com/ascii/ascii-example.odt">odt</a>, <a href="http://files.mpastell.com/ascii/ascii-example.html">html</a>, <a href="http://files.mpastell.com/ascii/ascii-example1.pdf">pdf</a> from latex and <a href="http://files.mpastell.com/ascii/ascii-example.pdf">pdf</a> from rst2pdf. You’ll need to have <a href="http://docutils.sourceforge.net/index.html">Docutils</a> and <a href="http://www.rexx.com/~dkuhlman/odtwriter.html">odtwriter</a> for the conversion.</p> <p>I have used the default options, but there are a lot options that can be passed to the docutils writers to customize the output, such as custom stylesheets. The image format in the pdfs is not optimal, but in can be changed with the “res” option in code chunk or you can also choose to use pdf format instead of jpg, but then it won’t show up in html documents. I think using reST with ascii package is a good option for producing reports and tutorials in multiple output formats. That is also because I’m already familiar with reST directives and also use Sphinx for other purposes too. The reST syntax is also very fast to learn and I think it is definitely worth exploring. I also recommend using Sphinx if you only want to get html and pdf output, because it has more directives than plain reST e.g. math support.</p> <div class="footnotes"> <hr /> <ol> <li id="fn1"><p>I changed the driver to output code chunks in reST literal environment <code>::</code> instead of <code>.. codeblock:: r</code>, because the codeblock directive is not supported in odt conversion and very poorly supported in latex conversion.<a href="#fnref1">↩</a></p></li> </ol> </div> tag:mpastell.com,2010-03-03:/2010/03/03/pweave-sweave-for-python/ Pweave - Sweave for Python 2010-03-02T22:00:00Z 2010-03-02T22:00:00Z <p>After looking into different options for literate programming with Python I ended up writing my own tool and call it <strong>Pweave</strong>. I have tried to implement the functionality of Sweave, that I really like and use for my R projects. Pweave is a single python script that is able to weave a python code between <code>&lt;&lt;&gt;&gt;=</code> and <code>@</code> blocks and include the results in the document. It can be used with reStructured text and Latex documents.</p> <p><strong>Update:</strong>. I’ve set up a <a href="http://mpastell.com/pweave">Pweave website</a>, you can also download it <a href="http://files.mpastell.com/Pweave">here</a>.</p> <p><strong>Codeblock options</strong> Pweave currently allows following options for the codeblocks (e.g. &lt;&gt;=): <em>fig</em> True or False. Whether a matplotlib plot produced in the codeblock should be included in the file. <em>echo</em> True or False. Echo the python code in the codeblock? <em>eval</em> True or False. Should the codeblock be evaluated? <em>results</em> “rst”, “tex” or “verbatim”. The output format of the printed results. <strong>Example:</strong> The syntax of Pweave is similar to Sweave, here is a small example sourcefile <a href="http://files.mpastell.com/ma.Pnw">ma.Pnw</a> which can be processed with Pweave using:</p> <div class="sourceCode"><pre class="sourceCode bash"><code class="sourceCode bash"><span class="kw">Pweave</span> ma.Pnw</code></pre></div> <p>which produces a reStructured text document <a href="http://files.mpastell.com/ma.rst">ma.rst</a>, which can in turn be processed to <a href="http://files.mpastell.com/ma.pdf">ma.pdf</a> with rst2latex and pdflatex.</p>