\documentclass[11pt]{article}
\parindent 0pt

\usepackage{fullpage}
\usepackage{psfig}

\begin{document}

\thispagestyle{empty}

\begin{center}
\Huge\bf
Homework \#\,\,5\\
\Large\bf
15-486/782: Artificial Neural Networks\\
Dave Touretzky, Fall 2006\\[.2in]
\end{center}


Assigned November 6, 2006.  Due November 13, 2006.\\[10pt]

\centerline{\psfig{file=hw5-net.eps,width=3in}}
$\quad$

Consider the Hopfield network shown above; it has four units and five
connections.  Assume the units use $+1/-1$ states.  We can denote
a state $\alpha$ of the network as a whole as a four-tuple
$\langle S_A,S_B,S_C,S_D\rangle$.

\begin{enumerate}

\item By hand, calculate the energy of the state $\langle -1, +1,+1,-1 \rangle$.
Show all your work.

\item Calculate the energies of the eight states where B has the value $+1$.
You can either do this by hand, or write code to do it for you.  Show
each state and its energy.

\item Now that you know the energies of eight of the network's sixteen 
states, how can you find the energies of the remaining eight states
without doing any more additions or multiplications?  (Hint: what do
you know about ``reversal states'' in Hopfield nets?)

\item In a Boltzmann machine at equilibrium, the probabilities of the states
satisfy the Boltzmann distribution:

$$\frac{P(\alpha)}{P(\beta)} = \frac{e^{-E_\alpha/T}}{e^{-E_\beta/T}}$$

From this relationship, show how to derive the formula for $P(\alpha)$
given the energies of all the states.  Hint: note that $\sum_\beta
P(\beta)$ must be 1.

\item Suppose we want to use stochastic units, i.e., make the network a
Boltzmann machine instead of a Hopfield net.  Given the energies of
all 16 states, and a temperature $T$, calculate the probability
$P(\alpha)$ of each state $\alpha$.  Do this for three temperatures:
$100$, $5$, and $0.1$.

Display your results as an array called \verb_Probabilities_ with 16
rows and 3 columns.  Also plot the result, and hand in the plot.  

The simplest way to plot the result is to write
{}\verb_plot(Probabilities)_.  However, another cool way to look at
the result is \verb_bar(Probabilities,'stacked')_.  Use whichever
one you like best.  Use the \verb_legend_ command to label your plot.

\item It's possible to add bias connections to a Hopfield net.  Let
each unit have a bias connection $b_i$, so that its net input is:

$$net_i = b_i + \sum_j w_{ij}s_j$$

You can think of the bias connection as just an ordinary connection to
an anonymous unit whose output is fixed at $+1$.  Based on this idea,
write down the energy function $E$ for a Hopfield net with bias
connections.

\item
Assume a $+1/-1$ Hopfield net with states $s_i$ and weights $w_{ij}$.
Derive a $1/0$ Hopfield net with states $S_i$, weights $W_{ij}$, and
bias connections $B_i$ that has the same energy landscape, i.e., the
same energy values for corresponding states.  Note: you will need to
add a constant term $C$ to the energy function.

\end{enumerate}

\end{document}