Understanding and Coding a neural network for XOR logic classifier from scratch by Shayan Ali Bhatti Analytics Vidhya

Figure 4 is the demonstration of the optimal value of scaling factor ‘b’. The XOR problem with neural networks can be solved by using Multi-Layer Perceptrons or a neural network architecture with an input layer, hidden layer, and output layer. So during the forward propagation through the neural networks, the weights get updated to the corresponding layers and the XOR logic gets executed. The Neural network architecture to solve the XOR problem will be as shown below.

Logistic regression is more reliable and can deal with a broader range of problems because it is based on probabilities and can model non-linear decision boundaries. But the perceptron model may be easier to use and use less computing power in some situations, especially when dealing with data that can be separated linearly. However, a single perceptron cannot model the XOR gate, which is not linearly separable. Instead, a multi-layer perceptron or a combination of perceptrons must be used to solve the XOR problem [5].

Though there are many kinds of activation functions, we’ll be using a simple linear activation function for our perceptron. The linear activation function has no effect on its input and outputs it as is. In a normal situation a cost function of 0 would mean high bias towards the training data and would be a problem.

It has utilized the nonlinearity of synapses to improve the capability of artificial neurons. XOR is a classical problem in the artificial neural network (ANN) [18]. The digital two-input XOR problem is represented in Figure 1. It is obvious here that the classes in two-dimensional XOR data distribution are the areas formed by two of the axes ‘X1’ and ‘X2’ (Here, X1 is input 1, and X2 is input 2). Furthermore, these areas represent respective classes simply by their sign (i.e., negative area corresponds to class 1, positive area corresponds to class 2).

To speed things up with the beauty of computer science – when we run this iteration 10,000 times, it gives us an output of about $.9999$. This is very close to our expected value of 1, and demonstrates that the network has learned what the correct output should be. It abruptely falls towards a small value and over epochs it slowly decreases. Following the development proposed by Ian Goodfellow et al, let’s use the mean squared error function (just like a regression problem) for the sake of simplicity.

The directive equation of a straight line, simple linear regression, math, cost functions

This blog comprehensively explores the perceptron model, its mathematics, binary classification, and logic gate generation applications. The perceptron model laid the foundation for deep learning, a subfield of machine learning focused on neural networks with multiple layers (deep neural networks). We have examined the performance of our proposed model for higher dimensional parity problems. It is to assess the applicability and generalization of our model. We have randomly varied the input dimension from 2 to 25 and compared the performance of our model with πt-neuron.

On the contrary, the function drawn to the right of the ReLU function is linear. Applying multiple linear activation functions xor neural network will still make the network linear. Adding more layers or nodes gives increasingly complex decision boundaries.

Generate 3D models based on descriptions

Therefore, a generalized solution is still required to solve these issues of the previous model. In this paper, we have suggested a generalized model for solving the XOR and higher-order parity problems by enhancing the pt-neuron model. To overcome the issue of the πt-neuron model, we have proposed an enhanced translated multiplicative model neuron (πt-neuron) model in this paper.

• Therefore, a generalized solution is still required to solve these issues of the previous model.
• Hidden layers are those layers with nodes other than the input and output nodes.
• This plot code is a bit more complex than the previous code samples but gives an extremely helpful insight into the workings of the neural network decision process for XOR.
• Ashutosh Mishra provided the resources and prepared the manuscript.
• As deep learning keeps improving, the perceptron model’s core ideas and principles will likely stay the same and influence the design of new architectures and algorithms.

So now let us understand how to solve the XOR problem with neural networks. The issue of vanishing gradient and nonconvergence in the previous πt-neuron model has been resolved by our proposed neuron model. It is because of the input dimension-dependent adaptable scaling factor (given in equation (6)). The effect of the scaling factor is already discussed in the previous section (as depicted in Figure 2(b)). We have seen that a larger scaling factor supports BP and results from proper convergence in the case of higher dimensional input. The significance of scaling has already been demonstrated in Figure 2(b).

There are no fixed rules on the number of hidden layers or the number of nodes in each layer of a network. The best performing models are obtained through trial and error. Remember the linear activation function we used on the output node of our perceptron model? You may have heard of the sigmoid and the tanh functions, which are some of the most popular non-linear activation functions. The perceptron model and logistic regression choices depend on the problem and dataset.

NeuralNetwork

We can plot the hyperplane separation of the decision boundaries. The sigmoid is a smooth function so there is no discontinuous boundary, rather we plot the transition from True into False. There are large regions of the input space which are mapped to an extremely small range. In these regions of the input space, even a large change will produce a small change in the output.

This is possible in our model by providing the compensation to each input (as given in our proposed enhanced πt-neuron model by equation (6)). We have considered the input distribution similar to Figure 5 (i.e., the input varies between [0, 1]) for each dimension. Results show that the effective scaling factor depends upon the dimension of input as well as the magnitude of the input.

This data is the same for each kind of logic gate, since they all take in two boolean variables as input. Apart from the usual visualization ( matplotliband seaborn) and numerical libraries (numpy), we’ll use cycle from itertools . This is done since our algorithm cycles through our data indefinitely until it manages to correctly classify the entire training data without any mistakes in the middle. The perceptron basically works as a threshold function — non-negative outputs are put into one class while negative ones are put into the other class. There are many other neural network architectures that can be trained to predict XOR, this is just one simple example. In the above code, the PyTorch library ‘functional’ containing the sigmoid function is imported.

In CNNs, the idea of weighted input signals and activation functions from perceptrons is carried over to the convolutional layers. To learn about spatial hierarchies in the data, these layers apply filters to the input regions near them. In the same way, RNNs build on the perceptron model by adding recurrent connections. This lets the network learn temporal dependencies in sequential data [25].

Some saw this new technology as essential for intelligent machines—a model for learning and changing [3]. Table 5 provide values of the threshold obtained by both the pt-neuron model and proposed models. In experiment #2 and experiment #3, the pt-neuron model has predicted threshold values beyond the range of inputs, i.e., [0, 1]. This is because we have not placed any limit on the values of the trainable parameter. It only reflects that the πt-neuron model has been unable to obtain the desired value in these experiments. Backpropagation is a way to update the weights and biases of a model starting from the output layer all the way to the beginning.

Nonetheless, with a solid foundation provided by this tutorial, you are well-equipped to tackle the challenges and opportunities in your journey through artificial intelligence. We also compared perceptrons and logistic regression, highlighting the differences and similarities by examining the role of a perceptron as a foundation for more advanced techniques in ML. We extended this upon setting perceptron’s role in artificial intelligence, historical significance, and ongoing influence. Whereη is the learning rate, a small positive constant that controls the step size of the updates. The above equation, along with the step function for its output, is activated (i.e., turned off via 0 or on via 1), as depicted in the following figure, Fig.

So the Class 0 region would be filled with the colour assigned to points belonging to that class. If not, we reset our counter, update our weights and continue the algorithm. Here, we cycle through the data indefinitely, keeping track of how many consecutive datapoints we correctly classified. If we manage to classify everything in one stretch, we terminate our algorithm. I will cover what an XOR is in this article, so there aren’t any prerequisites for reading this article.

Therefore in our loop we can do something like below to exit early. We should also store some of the variables and gradients to ensure convergence. Later we will require the derivative of this function so we can add in a factor of 0.5 which simplifies the derivative. Neurons fires a 1 if there is enough build up of voltage else it doesn’t fire (i.e a zero). The linear separable data points appear to be as shown below. The libraries used here like NumPy and pyplot are the same as those used in the Perceptron class.

Like I said earlier, the random synaptic weight will most likely not give us the correct output the first try. So we need a way to adjust the synpatic weights until it starts producing accurate https://traderoom.info/ outputs and “learns” the trend. The further $x$ goes in the positive direction, the closer it gets to 1. The further $x$ goes in the negative direction, the closer it gets to 0.

Let us understand why perceptrons cannot be used for XOR logic using the outputs generated by the XOR logic and the corresponding graph for XOR logic as shown below. We get our new weights by simply incrementing our original weights with the computed gradients multiplied by the learning rate. If either one of the bits is positive, then the result is positive.