This repository demonstrates feature selection using Particle Swarm Optimization (PSO) and Genetic Algorithm (GA) techniques on classic machine learning datasets. It includes comparison outputs and notebooks that show how feature subsets are selected and how classification performance changes before and after selection.
Feature selection is an important step in machine learning to reduce dimensionality, remove irrelevant features, and improve model generalization. This project applies:
- Particle Swarm Optimization (PSO)
- Genetic Algorithm (GA)
- Traditional feature selectors such as RFE, LASSO, and PCA
The main dataset used in the notebooks is the Breast Cancer Wisconsin dataset from sklearn.datasets.
FS_comp.txt- comparison output showing feature indices selected by RFE, LASSO, PCA, PSO, and GA.feature selection algorithms/PSO/PSO_FeatureSelection_pararell.ipynb- PSO-based feature selection with parallel execution.GA/(R)_data(cancer).ipynb- GA-based feature selection using the breast cancer dataset.GA/(KNN)_data(cancer).ipynb- GA-based selection evaluated with KNN.GA/(LR)_data(cancer).ipynb- GA-based selection evaluated with Logistic Regression.GA/(R)_data(Wine).ipynb- GA-based selection on the Wine dataset.
- Loads the breast cancer dataset.
- Applies MinMax scaling and splits into train/test sets.
- Defines a binary PSO algorithm for feature subset selection.
- Uses
RandomForestClassifierto evaluate candidate subsets. - Runs PSO in parallel across multiple independent runs.
- Aggregates selected features and ranks feature importance.
- Loads the breast cancer dataset and splits it into train/test sets.
- Initializes a binary GA population representing feature inclusion/exclusion.
- Defines a fitness function that trains a
RandomForestClassifierand minimizes1 - accuracy. - Uses selection, crossover, mutation, and generation replacement.
- Tracks best feature subsets and reports classifier accuracy before and after feature selection.
- Compares performance for a set of classifiers, including:
- Random Forest
- KNN
- SVM
- Logistic Regression
Based on the repository comparison output in FS_comp.txt:
- RFE selected feature indices:
[7, 3, 21, 20, 22] - LASSO selected feature indices:
[7, 20, 21, 27, 28] - PCA selected feature indices:
[7, 6, 27, 5, 22] - PSO selected feature indices:
[7, 3, 21, 27, 28] - GA selected feature indices:
[7, 20, 21, 8, 5]
For the GA breast cancer notebook, the feature selection process improves classifier performance in several cases, for example:
- Random Forest accuracy before selection:
0.9649 - Random Forest accuracy after selection:
0.9737 - KNN accuracy before selection:
0.9561 - KNN accuracy after selection:
0.9474 - SVM accuracy before selection:
0.9561 - SVM accuracy after selection:
0.9737 - Logistic Regression accuracy before selection:
0.9561 - Logistic Regression accuracy after selection:
0.9737
- Best Features Overall (Binary Representation):
[0 1 1 1 1 1 1 0 0 1 1 0 1 1 0 1 0 0 1 0 1 0 1 1 0 1 1 0 0 1] - Fitness of Best Features Overall:
0.06140350877192979
| Algorithm | Scenario | Accuracy | Time Taken |
|---|---|---|---|
| Random Forest | Before Feature Selection | 0.9649 | 0.3770 seconds |
| Random Forest | After Feature Selection | 0.9737 | 0.3303 seconds |
| KNN | Before Feature Selection | 0.9561 | 0.0090 seconds |
| KNN | After Feature Selection | 0.9474 | 0.0157 seconds |
| SVM | Before Feature Selection | 0.9561 | 1.3561 seconds |
| SVM | After Feature Selection | 0.9737 | 1.0043 seconds |
| Logistic Regression | Before Feature Selection | 0.9561 | 0.0075 seconds |
| Logistic Regression | After Feature Selection | 0.9737 | 0.0065 seconds |
- Total Selected Features:
18
Selected features (by index): 1, 2, 3, 4, 5, 6, 9, 10, 12, 13, 15, 18, 20, 22, 23, 25, 26, 29
Not selected features (by index): 0, 7, 8, 11, 14, 16, 17, 19, 21, 24, 27, 28
The project includes the following images in feature selection algorithms/IMG:
feature selection algorithms/IMG/GA_R_Cancer_Dataset.pngfeature selection algorithms/IMG/Best Fitness Values over PSO Iterations.pngfeature selection algorithms/IMG/Feature Importance Based on PSO Selection.png
For PSO results, the image explanations are already embedded inside the images themselves, so no additional textual description is required.
Install the required Python packages:
pip install numpy scikit-learn matplotlib tabulate ipythonIf you run the notebooks in Jupyter, also install:
pip install notebook jupyterlab- Open the repository folder in VS Code or Jupyter Lab.
- Open the notebook you want to run:
feature selection algorithms/PSO/PSO_FeatureSelection_pararell.ipynbfeature selection algorithms/GA/(R)_data(cancer).ipynb
- Execute cells sequentially.
- Review output tables and graphs for feature rankings, fitness progress, and accuracy comparisons.
- The PSO notebook uses parallel execution with
ThreadPoolExecutorto run multiple swarm optimization experiments concurrently. - The GA notebooks are useful templates for experimenting with different classifiers and mutation/crossover hyperparameters.
- The project is designed to compare metaheuristic feature selection against traditional methods.
You can extend this repository by:
- Adding more datasets (e.g., Wine, Diabetes, or custom CSV data).
- Comparing additional feature selection methods.
- Introducing cross-validation in the fitness function.
- Exploring different classifiers or ensemble models.
- Tuning PSO and GA hyperparameters for better results.
This README provides a complete guide for understanding and running the feature selection experiments in this repository.





