Hi,
I am doing a classification project by using matlab nn. I calculate the accuracy by the number of correct classified dividing by total number of instances. for the same data, i found the accuray varied significantly every time. It varied from 0.45 to 0.68. Why the results is so different? Are there any other ways to evaluate the accuracy of the results?
Hope somebody can help me.
Thanks.

Depending upon how the neural net is set up (size and number of inputs and number of traning epochs), accuracy can vary from numbers such as .45 to .68. Neural nets have a lot of flexibility, and with that comes a wide range of possibilities of what you'll get in terms of accuracy. Also, your network seems to be inaccurate if it's getting .45 or .68 for accuracy. You may want to adjust the size of your network so that it is more likely to find or get to the right answer and acheive greater accuracy.

The Intellector

When you say that the neural network's accuracy varied "over time", what precisely do you mean? Over several training iterations? What?

You say that you are measuring the accuracy of your neural network, but what criteria is being used to train it? Squared error? Cross-entropy?

How many classes are you predicting? Accuracy of 45% is worse than random if there are only two categories- one might as well choose the opposite of the class being predicted!

-Will Dwinnell
http://will.dwinnell.com

Hi,
This is a two valued classifiction, witch is either 0 or 1. I create a two layer network with 20,1 units, the first transfer function is tansig, next one is logsig,the performance function is mae, the other parameters of the network is default (which epoch is 100), then I use the training set and validation set to train the network, then I sim net by applying the validatin set and I compare the result with the original set, if they conform, then I add 1. Finally I divide the number of correct classification by the total number of the validation set, that is the accuracy.
Can you help me see what's wrong with it?
Thank you very much.

One major problem I see here is that you have no hidden layers. Hidden layers allow the network to adjust more and more (the more layers the more it can adjust) and in different ways. However, it can take longer for it to learn (it has to adjust more weights total). For a simple example of how different a single layer NN is from a multi-layer NN, try and have a single layer NN (meaning only input and output layers) learn the XOR logic gate function. It can't be done no matter what you do. However, add a hidden layer and it can be done. This is because the hidden layer allows the inputs to adjust in a different manner before it reaches the output layer. So, using the XOR example, a single layer would be like this:

Input layer takes in the two binary numbers (0, or 1), and applies the necessary weights. In order for the network to make sure that an input of 01, or 10 is 1, the weights have to both be above the threshold. So if the threshold is .5, the weights each must be .5 or higher. Now, with XOR, an input of 11 outputs a 0. Since the weights are .5 each, this can't be accomplished while maintaining the ability to output 1 with 01 or 10.

A hidden layer would work like this, for XOR:

Input layer layer takes in the two binary numbers and applies a set of necessary weights. Lets say that the input layer processes the output of 1 side of the issue (meaning inputs of 10, and 01, output 1, as well as 11 since the weights have no choice but to both be higher than the threshold). Now, that is sent to the hidden layer. Let's also say that a neuron in the input layer sends an output of 1 if the inputs were 11. Now, the hidden layer processes the 01/10 issue and the 11 issue. The 11 neuron (the one in the input layer that outputs 1 whenever 11 is input) will have a negative weight connected to the hidden layer neuron. The other input layer neuron will have a positive weight connected to the hidden layer neuron that exceeds its threshold (so the weights are -.2 (at least) and .6 (at least)). If both input layer neurons output a 1, the hidden layer neuron doesn't fire. If only one input layer neuron fires (which will always be the 01/10 neuron unless 00 is input), the hidden layer neuron will fire. So, in the end, the input layer tells the hidden layer the original input and the 01/10 processed input, and the hidden layer takes that information and completes the 00/11 input issue if it applies. The hidden layer then sends that to the output layer which outputs the correct XOR system.

As you may see, hidden layers can allow the inputs to be altered in ways that allow the network to acheive the correct response. The more hidden layers, the more the inputs can be altered, however, traning may take longer. Try using hidden layers to acheive your results.

The Intellector

Thank you for your reply.
Actually I tried 1 and 2 hidden layers network structure, the results are the same: in both situation, the accuracy varied largely.
So I think maybe I should average these results as a final result.

Your final results are going to be based on the test set, right?
So, which neural net are you going to use when you check the results of the test set?

"Hidden layers allow the network to adjust more and more (the more layers the more it can adjust) and in different ways."

While it is true that a MLP (multilayer perceptron) neural network using sigmoid transfer functions thoughout, with a single hidden layer can learn things which one without a hidden layer cannot, adding further layers does not change the representational power of the neural network. While it may be necessary to add neurons to a neural network with a single hidden layer to achieve the same results, there is no function which can be represented using two or more hidden layers which could not be represented using a single hidden layer.

-Predictor
http://will.dwinnell.com

True. As I recall, it was said that a network with 1 hidden layer can learn any function (almost, there's always something it can't learn).

The Intellector